Aug 04 10:25:37.461147 ip-10-0-142-113 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Aug 04 10:25:37.461162 ip-10-0-142-113 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Aug 04 10:25:37.461173 ip-10-0-142-113 systemd[1]: kubelet.service: Failed with result 'resources'. Aug 04 10:25:37.461607 ip-10-0-142-113 systemd[1]: Failed to start Kubernetes Kubelet. Aug 04 10:25:47.526693 ip-10-0-142-113 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Aug 04 10:25:47.526709 ip-10-0-142-113 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot fcad4201c9494cb0a09f703f53175d63 -- Aug 04 10:27:39.075096 ip-10-0-142-113 systemd[1]: Starting Kubernetes Kubelet... Aug 04 10:27:39.468076 ip-10-0-142-113 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 10:27:39.468076 ip-10-0-142-113 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Aug 04 10:27:39.468076 ip-10-0-142-113 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 10:27:39.468076 ip-10-0-142-113 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Aug 04 10:27:39.468076 ip-10-0-142-113 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 10:27:39.469104 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.468956 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Aug 04 10:27:39.473277 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473252 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 10:27:39.473277 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473275 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 10:27:39.473277 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473279 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 10:27:39.473277 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473282 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 10:27:39.473411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473285 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 10:27:39.473411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473288 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 10:27:39.473411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473292 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 10:27:39.473411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473295 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 10:27:39.473411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473298 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 10:27:39.473411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473300 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 10:27:39.473411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473303 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 10:27:39.473411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473306 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 10:27:39.473411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473310 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 10:27:39.473411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473314 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 10:27:39.473411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473318 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 10:27:39.473411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473321 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 10:27:39.473411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473329 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 10:27:39.473411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473332 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 10:27:39.473411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473334 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 10:27:39.473411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473337 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 10:27:39.473411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473340 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 10:27:39.473411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473342 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 10:27:39.473411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473345 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 10:27:39.473411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473348 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 10:27:39.473918 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473350 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 10:27:39.473918 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473353 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 10:27:39.473918 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473356 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 10:27:39.473918 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473358 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 10:27:39.473918 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473361 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 10:27:39.473918 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473363 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 10:27:39.473918 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473366 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 10:27:39.473918 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473368 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 10:27:39.473918 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473371 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 10:27:39.473918 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473374 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 10:27:39.473918 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473376 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 10:27:39.473918 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473378 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 10:27:39.473918 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473381 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 10:27:39.473918 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473386 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 10:27:39.473918 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473391 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 10:27:39.473918 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473394 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 10:27:39.473918 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473396 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 10:27:39.473918 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473399 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 10:27:39.473918 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473402 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 10:27:39.474395 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473405 2580 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 10:27:39.474395 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473407 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 10:27:39.474395 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473410 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 10:27:39.474395 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473413 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 10:27:39.474395 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473415 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 10:27:39.474395 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473418 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 10:27:39.474395 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473421 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 10:27:39.474395 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473424 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 10:27:39.474395 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473426 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 10:27:39.474395 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473429 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 10:27:39.474395 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473431 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 10:27:39.474395 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473434 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 10:27:39.474395 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473436 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 10:27:39.474395 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473439 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 10:27:39.474395 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473441 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 10:27:39.474395 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473443 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 10:27:39.474395 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473446 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 10:27:39.474395 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473448 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 10:27:39.474395 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473451 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 10:27:39.474395 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473454 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 10:27:39.474909 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473456 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 10:27:39.474909 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473459 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 10:27:39.474909 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473462 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 10:27:39.474909 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473464 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 10:27:39.474909 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473467 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 10:27:39.474909 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473469 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 10:27:39.474909 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473474 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 10:27:39.474909 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473477 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 10:27:39.474909 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473479 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 10:27:39.474909 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473482 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 10:27:39.474909 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473484 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 10:27:39.474909 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473487 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 10:27:39.474909 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473489 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 10:27:39.474909 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473492 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 10:27:39.474909 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473494 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 10:27:39.474909 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473497 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 10:27:39.474909 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473500 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 10:27:39.474909 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473505 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 10:27:39.474909 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473507 2580 feature_gate.go:328] unrecognized feature gate: Example Aug 04 10:27:39.475414 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473510 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 10:27:39.475414 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473512 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 10:27:39.475414 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473515 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 10:27:39.475414 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473517 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 10:27:39.475414 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473521 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 10:27:39.475414 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473591 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 10:27:39.475414 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473595 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 10:27:39.475414 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473599 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 10:27:39.475414 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473602 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 10:27:39.475414 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473605 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 10:27:39.475414 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473608 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 10:27:39.475414 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473611 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 10:27:39.475414 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473613 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 10:27:39.475414 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473616 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 10:27:39.475414 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473620 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 10:27:39.475414 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473623 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 10:27:39.475414 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.473625 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 10:27:39.475414 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474405 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 10:27:39.475414 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474413 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 10:27:39.475414 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474419 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 10:27:39.475924 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474422 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 10:27:39.475924 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474425 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 10:27:39.475924 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474428 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 10:27:39.475924 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474431 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 10:27:39.475924 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474433 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 10:27:39.475924 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474436 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 10:27:39.475924 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474438 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 10:27:39.475924 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474441 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 10:27:39.475924 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474443 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 10:27:39.475924 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474447 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 10:27:39.475924 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474452 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 10:27:39.475924 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474454 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 10:27:39.475924 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474457 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 10:27:39.475924 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474460 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 10:27:39.475924 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474462 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 10:27:39.475924 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474465 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 10:27:39.475924 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474467 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 10:27:39.475924 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474470 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 10:27:39.475924 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474473 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 10:27:39.476408 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474475 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 10:27:39.476408 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474477 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 10:27:39.476408 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474480 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 10:27:39.476408 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474483 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 10:27:39.476408 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474485 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 10:27:39.476408 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474488 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 10:27:39.476408 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474490 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 10:27:39.476408 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474493 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 10:27:39.476408 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474495 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 10:27:39.476408 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474498 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 10:27:39.476408 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474500 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 10:27:39.476408 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474503 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 10:27:39.476408 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474506 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 10:27:39.476408 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474508 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 10:27:39.476408 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474512 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 10:27:39.476408 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474514 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 10:27:39.476408 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474517 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 10:27:39.476408 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474519 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 10:27:39.476408 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474521 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 10:27:39.476408 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474524 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 10:27:39.476917 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474526 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 10:27:39.476917 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474529 2580 feature_gate.go:328] unrecognized feature gate: Example Aug 04 10:27:39.476917 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474531 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 10:27:39.476917 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474533 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 10:27:39.476917 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474536 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 10:27:39.476917 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474538 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 10:27:39.476917 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474541 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 10:27:39.476917 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474543 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 10:27:39.476917 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474546 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 10:27:39.476917 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474548 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 10:27:39.476917 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474551 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 10:27:39.476917 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474554 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 10:27:39.476917 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474557 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 10:27:39.476917 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474559 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 10:27:39.476917 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474561 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 10:27:39.476917 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474564 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 10:27:39.476917 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474567 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 10:27:39.476917 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474569 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 10:27:39.476917 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474571 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 10:27:39.476917 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474574 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 10:27:39.477411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474576 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 10:27:39.477411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474578 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 10:27:39.477411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474581 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 10:27:39.477411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474584 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 10:27:39.477411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474587 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 10:27:39.477411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474589 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 10:27:39.477411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474592 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 10:27:39.477411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474595 2580 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 10:27:39.477411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474597 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 10:27:39.477411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474600 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 10:27:39.477411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474602 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 10:27:39.477411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474604 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 10:27:39.477411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474606 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 10:27:39.477411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474609 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 10:27:39.477411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474611 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 10:27:39.477411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474614 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 10:27:39.477411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474616 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 10:27:39.477411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474618 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 10:27:39.477411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474621 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 10:27:39.477411 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474624 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 10:27:39.477955 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474626 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 10:27:39.477955 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474629 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 10:27:39.477955 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474631 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 10:27:39.477955 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474634 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 10:27:39.477955 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474636 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 10:27:39.477955 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474639 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 10:27:39.477955 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474641 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 10:27:39.477955 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474643 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 10:27:39.477955 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474646 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 10:27:39.477955 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474651 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 10:27:39.477955 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474655 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 10:27:39.477955 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474659 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 10:27:39.477955 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474662 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 10:27:39.477955 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474665 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 10:27:39.477955 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474669 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 10:27:39.477955 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474673 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 10:27:39.477955 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.474676 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 10:27:39.477955 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475081 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 10:27:39.477955 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475087 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 10:27:39.478424 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475091 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 10:27:39.478424 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475094 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 10:27:39.478424 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475096 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 10:27:39.478424 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475099 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 10:27:39.478424 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475101 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 10:27:39.478424 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475104 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 10:27:39.478424 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475107 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 10:27:39.478424 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475114 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 10:27:39.478424 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475116 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 10:27:39.478424 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475120 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 10:27:39.478424 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475124 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 10:27:39.478424 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475127 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 10:27:39.478424 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475129 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 10:27:39.478424 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475132 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 10:27:39.478424 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475134 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 10:27:39.478424 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475137 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 10:27:39.478424 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475139 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 10:27:39.478424 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475142 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 10:27:39.478424 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475145 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 10:27:39.478424 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475147 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 10:27:39.478936 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475149 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 10:27:39.478936 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475152 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 10:27:39.478936 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475155 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 10:27:39.478936 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475157 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 10:27:39.478936 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475159 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 10:27:39.478936 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475162 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 10:27:39.478936 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475165 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 10:27:39.478936 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475168 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 10:27:39.478936 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475171 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 10:27:39.478936 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475173 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 10:27:39.478936 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475177 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 10:27:39.478936 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475181 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 10:27:39.478936 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475183 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 10:27:39.478936 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475187 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 10:27:39.478936 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475190 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 10:27:39.478936 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475193 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 10:27:39.478936 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475196 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 10:27:39.478936 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475199 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 10:27:39.478936 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475201 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 10:27:39.478936 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475205 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 10:27:39.479419 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475208 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 10:27:39.479419 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475210 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 10:27:39.479419 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475213 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 10:27:39.479419 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475215 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 10:27:39.479419 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475217 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 10:27:39.479419 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475220 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 10:27:39.479419 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475222 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 10:27:39.479419 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475225 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 10:27:39.479419 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475227 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 10:27:39.479419 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475229 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 10:27:39.479419 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475232 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 10:27:39.479419 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475235 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 10:27:39.479419 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475237 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 10:27:39.479419 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475240 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 10:27:39.479419 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475242 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 10:27:39.479419 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475245 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 10:27:39.479419 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475247 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 10:27:39.479419 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475250 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 10:27:39.479419 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475252 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 10:27:39.479419 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475255 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 10:27:39.479927 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475258 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 10:27:39.479927 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475261 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 10:27:39.479927 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475263 2580 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 10:27:39.479927 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475266 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 10:27:39.479927 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475268 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 10:27:39.479927 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475271 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 10:27:39.479927 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475274 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 10:27:39.479927 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475277 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 10:27:39.479927 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475280 2580 feature_gate.go:328] unrecognized feature gate: Example Aug 04 10:27:39.479927 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475282 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 10:27:39.479927 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475284 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 10:27:39.479927 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475289 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 10:27:39.479927 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475291 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 10:27:39.479927 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475294 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 10:27:39.479927 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475296 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 10:27:39.479927 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475298 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 10:27:39.479927 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475302 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 10:27:39.479927 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475304 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 10:27:39.479927 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475307 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 10:27:39.480422 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475311 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 10:27:39.480422 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475314 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 10:27:39.480422 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475317 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 10:27:39.480422 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475320 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 10:27:39.480422 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475322 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 10:27:39.480422 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475325 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 10:27:39.480422 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475328 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 10:27:39.480422 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475330 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 10:27:39.480422 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475333 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 10:27:39.480422 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475335 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 10:27:39.480422 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475338 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 10:27:39.480422 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475340 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 10:27:39.480422 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475343 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 10:27:39.480422 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475346 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 10:27:39.480422 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475348 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 10:27:39.480422 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475351 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 10:27:39.480422 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475354 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 10:27:39.480422 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.475356 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 10:27:39.480422 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475439 2580 flags.go:64] FLAG: --address="0.0.0.0" Aug 04 10:27:39.480422 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475455 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Aug 04 10:27:39.480920 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475462 2580 flags.go:64] FLAG: --anonymous-auth="true" Aug 04 10:27:39.480920 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475467 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Aug 04 10:27:39.480920 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475471 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Aug 04 10:27:39.480920 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475474 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Aug 04 10:27:39.480920 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475479 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Aug 04 10:27:39.480920 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475484 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Aug 04 10:27:39.480920 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475487 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Aug 04 10:27:39.480920 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475490 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Aug 04 10:27:39.480920 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475494 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Aug 04 10:27:39.480920 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475497 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Aug 04 10:27:39.480920 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475500 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Aug 04 10:27:39.480920 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475503 2580 flags.go:64] FLAG: --cgroup-root="" Aug 04 10:27:39.480920 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475506 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Aug 04 10:27:39.480920 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475509 2580 flags.go:64] FLAG: --client-ca-file="" Aug 04 10:27:39.480920 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475512 2580 flags.go:64] FLAG: --cloud-provider="external" Aug 04 10:27:39.480920 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475515 2580 flags.go:64] FLAG: --cluster-dns="[]" Aug 04 10:27:39.480920 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475519 2580 flags.go:64] FLAG: --cluster-domain="" Aug 04 10:27:39.480920 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475522 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Aug 04 10:27:39.480920 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475525 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Aug 04 10:27:39.480920 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475528 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Aug 04 10:27:39.480920 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475532 2580 flags.go:64] FLAG: --container-log-max-files="5" Aug 04 10:27:39.480920 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475536 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Aug 04 10:27:39.480920 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475539 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Aug 04 10:27:39.481470 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475542 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Aug 04 10:27:39.481470 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475546 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Aug 04 10:27:39.481470 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475549 2580 flags.go:64] FLAG: --contention-profiling="false" Aug 04 10:27:39.481470 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475552 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Aug 04 10:27:39.481470 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475555 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Aug 04 10:27:39.481470 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475558 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Aug 04 10:27:39.481470 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475560 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Aug 04 10:27:39.481470 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475565 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Aug 04 10:27:39.481470 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475579 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Aug 04 10:27:39.481470 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475583 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Aug 04 10:27:39.481470 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475587 2580 flags.go:64] FLAG: --enable-load-reader="false" Aug 04 10:27:39.481470 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475590 2580 flags.go:64] FLAG: --enable-server="true" Aug 04 10:27:39.481470 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475593 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Aug 04 10:27:39.481470 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475598 2580 flags.go:64] FLAG: --event-burst="100" Aug 04 10:27:39.481470 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475602 2580 flags.go:64] FLAG: --event-qps="50" Aug 04 10:27:39.481470 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475605 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Aug 04 10:27:39.481470 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475608 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Aug 04 10:27:39.481470 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475611 2580 flags.go:64] FLAG: --eviction-hard="" Aug 04 10:27:39.481470 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475618 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Aug 04 10:27:39.481470 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475621 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Aug 04 10:27:39.481470 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475624 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Aug 04 10:27:39.481470 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475627 2580 flags.go:64] FLAG: --eviction-soft="" Aug 04 10:27:39.481470 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475630 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Aug 04 10:27:39.481470 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475633 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Aug 04 10:27:39.481470 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475636 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Aug 04 10:27:39.482270 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475638 2580 flags.go:64] FLAG: --experimental-mounter-path="" Aug 04 10:27:39.482270 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475641 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Aug 04 10:27:39.482270 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475644 2580 flags.go:64] FLAG: --fail-swap-on="true" Aug 04 10:27:39.482270 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475647 2580 flags.go:64] FLAG: --feature-gates="" Aug 04 10:27:39.482270 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475650 2580 flags.go:64] FLAG: --file-check-frequency="20s" Aug 04 10:27:39.482270 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475653 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Aug 04 10:27:39.482270 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475656 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Aug 04 10:27:39.482270 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475659 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Aug 04 10:27:39.482270 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475666 2580 flags.go:64] FLAG: --healthz-port="10248" Aug 04 10:27:39.482270 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475669 2580 flags.go:64] FLAG: --help="false" Aug 04 10:27:39.482270 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475672 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-142-113.ec2.internal" Aug 04 10:27:39.482270 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475675 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Aug 04 10:27:39.482270 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475678 2580 flags.go:64] FLAG: --http-check-frequency="20s" Aug 04 10:27:39.482270 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475681 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Aug 04 10:27:39.482270 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475684 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Aug 04 10:27:39.482270 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475688 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Aug 04 10:27:39.482270 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475691 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Aug 04 10:27:39.482270 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475694 2580 flags.go:64] FLAG: --image-service-endpoint="" Aug 04 10:27:39.482270 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475697 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Aug 04 10:27:39.482270 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475700 2580 flags.go:64] FLAG: --kube-api-burst="100" Aug 04 10:27:39.482270 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475703 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Aug 04 10:27:39.482270 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475706 2580 flags.go:64] FLAG: --kube-api-qps="50" Aug 04 10:27:39.482270 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475709 2580 flags.go:64] FLAG: --kube-reserved="" Aug 04 10:27:39.482270 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475713 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Aug 04 10:27:39.483357 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475715 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Aug 04 10:27:39.483357 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475718 2580 flags.go:64] FLAG: --kubelet-cgroups="" Aug 04 10:27:39.483357 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475721 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Aug 04 10:27:39.483357 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475724 2580 flags.go:64] FLAG: --lock-file="" Aug 04 10:27:39.483357 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475726 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Aug 04 10:27:39.483357 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475729 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Aug 04 10:27:39.483357 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475732 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Aug 04 10:27:39.483357 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475737 2580 flags.go:64] FLAG: --log-json-split-stream="false" Aug 04 10:27:39.483357 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475740 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Aug 04 10:27:39.483357 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475742 2580 flags.go:64] FLAG: --log-text-split-stream="false" Aug 04 10:27:39.483357 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475745 2580 flags.go:64] FLAG: --logging-format="text" Aug 04 10:27:39.483357 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475762 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Aug 04 10:27:39.483357 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475766 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Aug 04 10:27:39.483357 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475769 2580 flags.go:64] FLAG: --manifest-url="" Aug 04 10:27:39.483357 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475772 2580 flags.go:64] FLAG: --manifest-url-header="" Aug 04 10:27:39.483357 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475776 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Aug 04 10:27:39.483357 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475780 2580 flags.go:64] FLAG: --max-open-files="1000000" Aug 04 10:27:39.483357 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475785 2580 flags.go:64] FLAG: --max-pods="110" Aug 04 10:27:39.483357 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475788 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Aug 04 10:27:39.483357 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475791 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Aug 04 10:27:39.483357 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475794 2580 flags.go:64] FLAG: --memory-manager-policy="None" Aug 04 10:27:39.483357 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475797 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Aug 04 10:27:39.483357 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475800 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Aug 04 10:27:39.483357 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475803 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Aug 04 10:27:39.483357 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475806 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Aug 04 10:27:39.484445 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475814 2580 flags.go:64] FLAG: --node-status-max-images="50" Aug 04 10:27:39.484445 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475817 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Aug 04 10:27:39.484445 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475827 2580 flags.go:64] FLAG: --oom-score-adj="-999" Aug 04 10:27:39.484445 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475830 2580 flags.go:64] FLAG: --pod-cidr="" Aug 04 10:27:39.484445 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475833 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Aug 04 10:27:39.484445 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475839 2580 flags.go:64] FLAG: --pod-manifest-path="" Aug 04 10:27:39.484445 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475842 2580 flags.go:64] FLAG: --pod-max-pids="-1" Aug 04 10:27:39.484445 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475845 2580 flags.go:64] FLAG: --pods-per-core="0" Aug 04 10:27:39.484445 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475847 2580 flags.go:64] FLAG: --port="10250" Aug 04 10:27:39.484445 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475850 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Aug 04 10:27:39.484445 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475853 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-040e384b235377d6a" Aug 04 10:27:39.484445 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475856 2580 flags.go:64] FLAG: --qos-reserved="" Aug 04 10:27:39.484445 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475859 2580 flags.go:64] FLAG: --read-only-port="10255" Aug 04 10:27:39.484445 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475861 2580 flags.go:64] FLAG: --register-node="true" Aug 04 10:27:39.484445 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475864 2580 flags.go:64] FLAG: --register-with-taints="" Aug 04 10:27:39.484445 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475867 2580 flags.go:64] FLAG: --registry-burst="10" Aug 04 10:27:39.484445 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475870 2580 flags.go:64] FLAG: --registry-qps="5" Aug 04 10:27:39.484445 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475873 2580 flags.go:64] FLAG: --reserved-cpus="" Aug 04 10:27:39.484445 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475876 2580 flags.go:64] FLAG: --reserved-memory="" Aug 04 10:27:39.484445 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475880 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Aug 04 10:27:39.484445 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475883 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Aug 04 10:27:39.484445 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475886 2580 flags.go:64] FLAG: --rotate-certificates="false" Aug 04 10:27:39.484445 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475888 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Aug 04 10:27:39.484445 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475891 2580 flags.go:64] FLAG: --runonce="false" Aug 04 10:27:39.484445 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475895 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Aug 04 10:27:39.485135 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475898 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Aug 04 10:27:39.485135 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475901 2580 flags.go:64] FLAG: --seccomp-default="false" Aug 04 10:27:39.485135 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475904 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Aug 04 10:27:39.485135 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475907 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Aug 04 10:27:39.485135 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475910 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Aug 04 10:27:39.485135 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475913 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Aug 04 10:27:39.485135 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475917 2580 flags.go:64] FLAG: --storage-driver-password="root" Aug 04 10:27:39.485135 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475920 2580 flags.go:64] FLAG: --storage-driver-secure="false" Aug 04 10:27:39.485135 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475922 2580 flags.go:64] FLAG: --storage-driver-table="stats" Aug 04 10:27:39.485135 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475925 2580 flags.go:64] FLAG: --storage-driver-user="root" Aug 04 10:27:39.485135 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475933 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Aug 04 10:27:39.485135 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475937 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Aug 04 10:27:39.485135 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475940 2580 flags.go:64] FLAG: --system-cgroups="" Aug 04 10:27:39.485135 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475942 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Aug 04 10:27:39.485135 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475948 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Aug 04 10:27:39.485135 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475950 2580 flags.go:64] FLAG: --tls-cert-file="" Aug 04 10:27:39.485135 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475953 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Aug 04 10:27:39.485135 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475957 2580 flags.go:64] FLAG: --tls-min-version="" Aug 04 10:27:39.485135 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475959 2580 flags.go:64] FLAG: --tls-private-key-file="" Aug 04 10:27:39.485135 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475963 2580 flags.go:64] FLAG: --topology-manager-policy="none" Aug 04 10:27:39.485135 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475965 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Aug 04 10:27:39.485135 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475968 2580 flags.go:64] FLAG: --topology-manager-scope="container" Aug 04 10:27:39.485135 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475971 2580 flags.go:64] FLAG: --v="2" Aug 04 10:27:39.485135 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475975 2580 flags.go:64] FLAG: --version="false" Aug 04 10:27:39.485135 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475979 2580 flags.go:64] FLAG: --vmodule="" Aug 04 10:27:39.486236 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475983 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Aug 04 10:27:39.486236 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.475986 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Aug 04 10:27:39.486236 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476140 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 10:27:39.486236 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476145 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 10:27:39.486236 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476148 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 10:27:39.486236 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476151 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 10:27:39.486236 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476154 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 10:27:39.486236 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476157 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 10:27:39.486236 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476160 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 10:27:39.486236 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476162 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 10:27:39.486236 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476165 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 10:27:39.486236 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476167 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 10:27:39.486236 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476170 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 10:27:39.486236 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476174 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 10:27:39.486236 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476176 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 10:27:39.486236 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476180 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 10:27:39.486236 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476183 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 10:27:39.486236 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476186 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 10:27:39.486236 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476189 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 10:27:39.486236 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476192 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 10:27:39.487026 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476195 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 10:27:39.487026 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476197 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 10:27:39.487026 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476200 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 10:27:39.487026 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476202 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 10:27:39.487026 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476205 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 10:27:39.487026 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476207 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 10:27:39.487026 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476209 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 10:27:39.487026 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476212 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 10:27:39.487026 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476215 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 10:27:39.487026 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476217 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 10:27:39.487026 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476219 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 10:27:39.487026 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476222 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 10:27:39.487026 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476224 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 10:27:39.487026 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476226 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 10:27:39.487026 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476230 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 10:27:39.487026 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476234 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 10:27:39.487026 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476237 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 10:27:39.487026 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476239 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 10:27:39.487026 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476243 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 10:27:39.487632 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476246 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 10:27:39.487632 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476249 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 10:27:39.487632 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476252 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 10:27:39.487632 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476255 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 10:27:39.487632 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476257 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 10:27:39.487632 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476260 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 10:27:39.487632 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476265 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 10:27:39.487632 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476268 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 10:27:39.487632 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476270 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 10:27:39.487632 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476273 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 10:27:39.487632 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476276 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 10:27:39.487632 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476278 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 10:27:39.487632 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476287 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 10:27:39.487632 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476290 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 10:27:39.487632 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476292 2580 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 10:27:39.487632 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476294 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 10:27:39.487632 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476297 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 10:27:39.487632 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476300 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 10:27:39.487632 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476304 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 10:27:39.487632 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476307 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 10:27:39.488485 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476310 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 10:27:39.488485 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476313 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 10:27:39.488485 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476315 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 10:27:39.488485 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476317 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 10:27:39.488485 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476320 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 10:27:39.488485 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476322 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 10:27:39.488485 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476324 2580 feature_gate.go:328] unrecognized feature gate: Example Aug 04 10:27:39.488485 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476327 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 10:27:39.488485 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476330 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 10:27:39.488485 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476332 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 10:27:39.488485 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476335 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 10:27:39.488485 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476339 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 10:27:39.488485 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476341 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 10:27:39.488485 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476344 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 10:27:39.488485 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476346 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 10:27:39.488485 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476349 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 10:27:39.488485 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476351 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 10:27:39.488485 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476354 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 10:27:39.488485 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476358 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 10:27:39.488485 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476361 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 10:27:39.489189 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476363 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 10:27:39.489189 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476366 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 10:27:39.489189 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476368 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 10:27:39.489189 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476371 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 10:27:39.489189 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476373 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 10:27:39.489189 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476381 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 10:27:39.489189 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476384 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 10:27:39.489189 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476386 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 10:27:39.489189 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476388 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 10:27:39.489189 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476391 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 10:27:39.489189 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476393 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 10:27:39.489189 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476396 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 10:27:39.489189 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476398 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 10:27:39.489189 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476401 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 10:27:39.489189 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476403 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 10:27:39.489189 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476406 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 10:27:39.489189 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476408 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 10:27:39.489189 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476411 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 10:27:39.489189 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476414 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 10:27:39.489189 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476417 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 10:27:39.489875 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476419 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 10:27:39.489875 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.476422 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 10:27:39.489875 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.476433 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 10:27:39.489875 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.484368 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Aug 04 10:27:39.489875 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.484395 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Aug 04 10:27:39.489875 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.485202 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Aug 04 10:27:39.489875 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.485225 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Aug 04 10:27:39.489875 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485329 2580 feature_gate.go:328] unrecognized feature gate: Example Aug 04 10:27:39.489875 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485339 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 10:27:39.489875 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485345 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 10:27:39.489875 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485350 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 10:27:39.489875 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485355 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 10:27:39.489875 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485359 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 10:27:39.489875 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485365 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 10:27:39.490278 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485369 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 10:27:39.490278 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485373 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 10:27:39.490278 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485378 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 10:27:39.490278 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485384 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 10:27:39.490278 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485389 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 10:27:39.490278 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485393 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 10:27:39.490278 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485397 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 10:27:39.490278 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485402 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 10:27:39.490278 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485406 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 10:27:39.490278 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485410 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 10:27:39.490278 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485414 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 10:27:39.490278 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485418 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 10:27:39.490278 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485422 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 10:27:39.490278 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485427 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 10:27:39.490278 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485431 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 10:27:39.490278 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485436 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 10:27:39.490278 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485440 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 10:27:39.490278 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485444 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 10:27:39.490278 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485449 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 10:27:39.490278 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485453 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 10:27:39.490808 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485457 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 10:27:39.490808 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485460 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 10:27:39.490808 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485466 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 10:27:39.490808 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485470 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 10:27:39.490808 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485478 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 10:27:39.490808 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485484 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 10:27:39.490808 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485488 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 10:27:39.490808 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485493 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 10:27:39.490808 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485498 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 10:27:39.490808 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485503 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 10:27:39.490808 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485507 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 10:27:39.490808 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485511 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 10:27:39.490808 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485516 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 10:27:39.490808 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485520 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 10:27:39.490808 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485524 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 10:27:39.490808 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485528 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 10:27:39.490808 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485535 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 10:27:39.490808 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485540 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 10:27:39.490808 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485544 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 10:27:39.491275 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485548 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 10:27:39.491275 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485552 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 10:27:39.491275 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485556 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 10:27:39.491275 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485560 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 10:27:39.491275 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485565 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 10:27:39.491275 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485569 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 10:27:39.491275 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485573 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 10:27:39.491275 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485577 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 10:27:39.491275 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485582 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 10:27:39.491275 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485585 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 10:27:39.491275 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485589 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 10:27:39.491275 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485593 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 10:27:39.491275 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485597 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 10:27:39.491275 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485602 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 10:27:39.491275 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485606 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 10:27:39.491275 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485611 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 10:27:39.491275 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485616 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 10:27:39.491275 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485620 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 10:27:39.491275 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485624 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 10:27:39.491275 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485628 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 10:27:39.491963 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485632 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 10:27:39.491963 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485636 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 10:27:39.491963 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485640 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 10:27:39.491963 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485645 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 10:27:39.491963 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485649 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 10:27:39.491963 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485653 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 10:27:39.491963 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485658 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 10:27:39.491963 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485662 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 10:27:39.491963 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485668 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 10:27:39.491963 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485674 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 10:27:39.491963 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485679 2580 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 10:27:39.491963 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485683 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 10:27:39.491963 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485687 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 10:27:39.491963 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485692 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 10:27:39.491963 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485696 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 10:27:39.491963 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485700 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 10:27:39.491963 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485704 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 10:27:39.491963 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485708 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 10:27:39.491963 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485712 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 10:27:39.491963 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485716 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 10:27:39.492505 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485720 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 10:27:39.492505 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485724 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 10:27:39.492505 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485728 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 10:27:39.492505 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485735 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 10:27:39.492505 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485740 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 10:27:39.492505 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485746 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 10:27:39.492505 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485769 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 10:27:39.492505 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485775 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 10:27:39.492505 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485780 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 10:27:39.492505 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485784 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 10:27:39.492505 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485788 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 10:27:39.492505 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485792 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 10:27:39.492505 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.485797 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 10:27:39.492505 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.485806 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 10:27:39.492505 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486047 2580 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 10:27:39.492908 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486055 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 10:27:39.492908 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486061 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 10:27:39.492908 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486065 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 10:27:39.492908 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486070 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 10:27:39.492908 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486074 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 10:27:39.492908 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486079 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 10:27:39.492908 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486084 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 10:27:39.492908 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486088 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 10:27:39.492908 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486093 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 10:27:39.492908 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486098 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 10:27:39.492908 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486102 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 10:27:39.492908 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486106 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 10:27:39.492908 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486110 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 10:27:39.492908 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486115 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 10:27:39.492908 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486119 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 10:27:39.492908 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486123 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 10:27:39.492908 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486127 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 10:27:39.492908 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486131 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 10:27:39.492908 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486136 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 10:27:39.492908 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486140 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 10:27:39.493524 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486144 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 10:27:39.493524 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486148 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 10:27:39.493524 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486153 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 10:27:39.493524 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486157 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 10:27:39.493524 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486162 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 10:27:39.493524 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486169 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 10:27:39.493524 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486176 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 10:27:39.493524 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486180 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 10:27:39.493524 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486185 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 10:27:39.493524 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486189 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 10:27:39.493524 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486193 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 10:27:39.493524 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486197 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 10:27:39.493524 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486201 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 10:27:39.493524 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486205 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 10:27:39.493524 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486209 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 10:27:39.493524 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486214 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 10:27:39.493524 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486218 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 10:27:39.493524 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486222 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 10:27:39.493524 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486226 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 10:27:39.494093 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486232 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 10:27:39.494093 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486237 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 10:27:39.494093 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486243 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 10:27:39.494093 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486247 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 10:27:39.494093 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486251 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 10:27:39.494093 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486255 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 10:27:39.494093 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486258 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 10:27:39.494093 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486262 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 10:27:39.494093 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486266 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 10:27:39.494093 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486270 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 10:27:39.494093 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486275 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 10:27:39.494093 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486279 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 10:27:39.494093 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486283 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 10:27:39.494093 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486288 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 10:27:39.494093 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486292 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 10:27:39.494093 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486296 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 10:27:39.494093 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486300 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 10:27:39.494093 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486304 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 10:27:39.494093 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486309 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 10:27:39.494846 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486313 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 10:27:39.494846 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486317 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 10:27:39.494846 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486321 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 10:27:39.494846 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486325 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 10:27:39.494846 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486330 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 10:27:39.494846 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486336 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 10:27:39.494846 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486342 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 10:27:39.494846 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486346 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 10:27:39.494846 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486351 2580 feature_gate.go:328] unrecognized feature gate: Example Aug 04 10:27:39.494846 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486355 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 10:27:39.494846 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486360 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 10:27:39.494846 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486364 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 10:27:39.494846 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486369 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 10:27:39.494846 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486373 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 10:27:39.494846 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486378 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 10:27:39.494846 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486383 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 10:27:39.494846 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486388 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 10:27:39.494846 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486392 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 10:27:39.494846 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486396 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 10:27:39.495401 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486401 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 10:27:39.495401 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486405 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 10:27:39.495401 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486409 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 10:27:39.495401 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486414 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 10:27:39.495401 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486418 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 10:27:39.495401 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486422 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 10:27:39.495401 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486426 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 10:27:39.495401 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486430 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 10:27:39.495401 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486434 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 10:27:39.495401 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486438 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 10:27:39.495401 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486442 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 10:27:39.495401 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486446 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 10:27:39.495401 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486451 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 10:27:39.495401 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486456 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 10:27:39.495401 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486460 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 10:27:39.495401 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486464 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 10:27:39.495401 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486468 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 10:27:39.495401 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486473 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 10:27:39.495401 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486477 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 10:27:39.495401 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486482 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 10:27:39.495937 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:39.486487 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 10:27:39.495937 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.486496 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 10:27:39.495937 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.486687 2580 server.go:962] "Client rotation is on, will bootstrap in background" Aug 04 10:27:39.495937 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.489544 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Aug 04 10:27:39.495937 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.491405 2580 server.go:1019] "Starting client certificate rotation" Aug 04 10:27:39.495937 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.491518 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 10:27:39.495937 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.491587 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 10:27:39.497232 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.497208 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 10:27:39.498536 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.498518 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 10:27:39.505109 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.505090 2580 log.go:25] "Validated CRI v1 runtime API" Aug 04 10:27:39.507189 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.507173 2580 log.go:25] "Validated CRI v1 image API" Aug 04 10:27:39.510451 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.510427 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Aug 04 10:27:39.512826 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.512806 2580 fs.go:135] Filesystem UUIDs: map[30538038-df0c-4d36-9ca1-0d8507a6903e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7d38efbd-3b81-465b-8273-a3cac054da0c:/dev/nvme0n1p3] Aug 04 10:27:39.512881 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.512826 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Aug 04 10:27:39.518606 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.518476 2580 manager.go:216] Machine: {Timestamp:2026-08-04 10:27:39.516525631 +0000 UTC m=+0.331221027 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3176721 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec230804d9ce16fe9a48632718eca92c SystemUUID:ec230804-d9ce-16fe-9a48-632718eca92c BootID:fcad4201-c949-4cb0-a09f-703f53175d63 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a1:d4:37:8c:b5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a1:d4:37:8c:b5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:2c:8e:6f:76:c7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Aug 04 10:27:39.518606 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.518588 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Aug 04 10:27:39.518737 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.518680 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Aug 04 10:27:39.520459 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.520439 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 10:27:39.521721 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.521694 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Aug 04 10:27:39.521887 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.521725 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-113.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Aug 04 10:27:39.522425 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.522416 2580 topology_manager.go:138] "Creating topology manager with none policy" Aug 04 10:27:39.522459 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.522429 2580 container_manager_linux.go:309] "Creating device plugin manager" Aug 04 10:27:39.522459 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.522436 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 10:27:39.522459 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.522451 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 10:27:39.523367 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.523357 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Aug 04 10:27:39.526376 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.526364 2580 state_mem.go:36] "Initialized new in-memory state store" Aug 04 10:27:39.526499 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.526489 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Aug 04 10:27:39.526539 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.526519 2580 kubelet.go:486] "Attempting to sync node with API server" Aug 04 10:27:39.526539 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.526530 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Aug 04 10:27:39.526618 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.526544 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Aug 04 10:27:39.526618 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.526553 2580 kubelet.go:398] "Adding apiserver pod source" Aug 04 10:27:39.526618 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.526565 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Aug 04 10:27:39.526967 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.526953 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 10:27:39.527026 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.526984 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 10:27:39.530440 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.530415 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Aug 04 10:27:39.532658 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.532641 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Aug 04 10:27:39.532738 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.532673 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Aug 04 10:27:39.534043 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.534031 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Aug 04 10:27:39.534096 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.534049 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Aug 04 10:27:39.534096 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.534056 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Aug 04 10:27:39.534096 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.534061 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Aug 04 10:27:39.534096 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.534070 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Aug 04 10:27:39.534096 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.534076 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Aug 04 10:27:39.534096 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.534084 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Aug 04 10:27:39.534096 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.534089 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Aug 04 10:27:39.534096 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.534096 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Aug 04 10:27:39.534304 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.534101 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Aug 04 10:27:39.534304 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.534110 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Aug 04 10:27:39.534304 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.534116 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Aug 04 10:27:39.535634 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.535618 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Aug 04 10:27:39.535634 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.535633 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Aug 04 10:27:39.539915 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:39.539887 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Aug 04 10:27:39.539996 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:39.539888 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-113.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Aug 04 10:27:39.542360 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.542347 2580 server.go:1268] "Started kubelet" Aug 04 10:27:39.542443 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.542412 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Aug 04 10:27:39.543238 ip-10-0-142-113 systemd[1]: Started Kubernetes Kubelet. Aug 04 10:27:39.543387 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.543214 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Aug 04 10:27:39.543387 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.543315 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Aug 04 10:27:39.543680 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.543663 2580 server.go:310] "Adding debug handlers to kubelet server" Aug 04 10:27:39.546243 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.546228 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Aug 04 10:27:39.547287 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.547271 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-113.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Aug 04 10:27:39.547965 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:39.547054 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-113.ec2.internal.18c8935842f75bba default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-113.ec2.internal,UID:ip-10-0-142-113.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-113.ec2.internal,},FirstTimestamp:2026-08-04 10:27:39.542313914 +0000 UTC m=+0.357009371,LastTimestamp:2026-08-04 10:27:39.542313914 +0000 UTC m=+0.357009371,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-113.ec2.internal,}" Aug 04 10:27:39.548559 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:39.548540 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Aug 04 10:27:39.549705 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.549687 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Aug 04 10:27:39.549803 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:39.549788 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-113.ec2.internal\" not found" Aug 04 10:27:39.549856 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.549687 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Aug 04 10:27:39.550231 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.550198 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Aug 04 10:27:39.550318 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.550230 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Aug 04 10:27:39.550318 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.550247 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Aug 04 10:27:39.550420 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.550329 2580 reconstruct.go:97] "Volume reconstruction finished" Aug 04 10:27:39.550420 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.550339 2580 reconciler.go:29] "Reconciler: start to sync state" Aug 04 10:27:39.550582 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.550563 2580 factory.go:55] Registering systemd factory Aug 04 10:27:39.550655 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.550585 2580 factory.go:223] Registration of the systemd container factory successfully Aug 04 10:27:39.550936 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.550922 2580 factory.go:153] Registering CRI-O factory Aug 04 10:27:39.550936 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.550937 2580 factory.go:223] Registration of the crio container factory successfully Aug 04 10:27:39.551019 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.550988 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Aug 04 10:27:39.551019 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.551007 2580 factory.go:103] Registering Raw factory Aug 04 10:27:39.551019 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.551019 2580 manager.go:1196] Started watching for new ooms in manager Aug 04 10:27:39.551358 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.551347 2580 manager.go:349] Starting recovery of all containers Aug 04 10:27:39.554324 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.554290 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t44ng" Aug 04 10:27:39.561139 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.561118 2580 manager.go:354] Recovery completed Aug 04 10:27:39.562166 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.562138 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t44ng" Aug 04 10:27:39.563124 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:39.563086 2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-113.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Aug 04 10:27:39.563563 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:39.563539 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Aug 04 10:27:39.567727 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.567713 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 10:27:39.569701 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.569680 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-113.ec2.internal" event="NodeHasSufficientMemory" Aug 04 10:27:39.569809 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.569713 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-113.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 10:27:39.569809 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.569723 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-113.ec2.internal" event="NodeHasSufficientPID" Aug 04 10:27:39.570288 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.570270 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 10:27:39.570367 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.570273 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Aug 04 10:27:39.570367 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.570363 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Aug 04 10:27:39.570450 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.570384 2580 state_mem.go:36] "Initialized new in-memory state store" Aug 04 10:27:39.571073 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.571056 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-113.ec2.internal" event="NodeHasSufficientMemory" Aug 04 10:27:39.571131 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.571089 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-113.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 10:27:39.571131 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.571099 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-113.ec2.internal" event="NodeHasSufficientPID" Aug 04 10:27:39.571512 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.571496 2580 policy_none.go:49] "None policy: Start" Aug 04 10:27:39.571588 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.571515 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Aug 04 10:27:39.571588 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.571526 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Aug 04 10:27:39.572527 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.572515 2580 policy_none.go:47] "Start" Aug 04 10:27:39.597544 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.597366 2580 manager.go:337] "Starting Device Plugin manager" Aug 04 10:27:39.597675 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:39.597593 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Aug 04 10:27:39.597675 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.597608 2580 server.go:87] "Starting device plugin registration server" Aug 04 10:27:39.597877 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.597864 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Aug 04 10:27:39.597936 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.597880 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Aug 04 10:27:39.597998 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.597980 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Aug 04 10:27:39.598330 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.598061 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Aug 04 10:27:39.598330 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.598074 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Aug 04 10:27:39.598967 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:39.598934 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Aug 04 10:27:39.599065 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:39.598988 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-113.ec2.internal\" not found" Aug 04 10:27:39.697836 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.697798 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Aug 04 10:27:39.698919 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.697948 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 10:27:39.698919 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.698903 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-113.ec2.internal" event="NodeHasSufficientMemory" Aug 04 10:27:39.699051 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.698935 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-113.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 10:27:39.699051 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.698949 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-113.ec2.internal" event="NodeHasSufficientPID" Aug 04 10:27:39.699051 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.698980 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-113.ec2.internal" Aug 04 10:27:39.699194 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.699158 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Aug 04 10:27:39.699248 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.699207 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Aug 04 10:27:39.699248 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.699231 2580 kubelet.go:2448] "Starting kubelet main sync loop" Aug 04 10:27:39.699351 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:39.699272 2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Aug 04 10:27:39.703305 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.703288 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 10:27:39.705514 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.705499 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-113.ec2.internal" Aug 04 10:27:39.705591 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:39.705520 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-113.ec2.internal\": node \"ip-10-0-142-113.ec2.internal\" not found" Aug 04 10:27:39.749612 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:39.749536 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-113.ec2.internal\" not found" Aug 04 10:27:39.799670 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.799618 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-113.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-113.ec2.internal"] Aug 04 10:27:39.799879 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.799722 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 10:27:39.800783 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.800767 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-113.ec2.internal" event="NodeHasSufficientMemory" Aug 04 10:27:39.800871 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.800797 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-113.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 10:27:39.800871 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.800809 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-113.ec2.internal" event="NodeHasSufficientPID" Aug 04 10:27:39.802172 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.802159 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 10:27:39.802296 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.802282 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-113.ec2.internal" Aug 04 10:27:39.802336 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.802318 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 10:27:39.802963 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.802945 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-113.ec2.internal" event="NodeHasSufficientMemory" Aug 04 10:27:39.803039 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.802974 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-113.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 10:27:39.803039 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.802985 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-113.ec2.internal" event="NodeHasSufficientPID" Aug 04 10:27:39.803039 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.802948 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-113.ec2.internal" event="NodeHasSufficientMemory" Aug 04 10:27:39.803140 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.803051 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-113.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 10:27:39.803140 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.803063 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-113.ec2.internal" event="NodeHasSufficientPID" Aug 04 10:27:39.804179 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.804165 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-113.ec2.internal" Aug 04 10:27:39.804222 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.804194 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 10:27:39.804933 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.804911 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-113.ec2.internal" event="NodeHasSufficientMemory" Aug 04 10:27:39.805007 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.804942 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-113.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 10:27:39.805007 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.804958 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-113.ec2.internal" event="NodeHasSufficientPID" Aug 04 10:27:39.829637 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:39.829604 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-113.ec2.internal\" not found" node="ip-10-0-142-113.ec2.internal" Aug 04 10:27:39.834167 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:39.834151 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-113.ec2.internal\" not found" node="ip-10-0-142-113.ec2.internal" Aug 04 10:27:39.849835 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:39.849811 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-113.ec2.internal\" not found" Aug 04 10:27:39.852053 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.852035 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3d393a7cd5ccc56f50081d0bd7ed54a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-113.ec2.internal\" (UID: \"e3d393a7cd5ccc56f50081d0bd7ed54a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-113.ec2.internal" Aug 04 10:27:39.852104 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.852061 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3d393a7cd5ccc56f50081d0bd7ed54a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-113.ec2.internal\" (UID: \"e3d393a7cd5ccc56f50081d0bd7ed54a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-113.ec2.internal" Aug 04 10:27:39.852104 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.852079 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1a9bc83f737210ec581d9649412c17cd-config\") pod \"kube-apiserver-proxy-ip-10-0-142-113.ec2.internal\" (UID: \"1a9bc83f737210ec581d9649412c17cd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-113.ec2.internal" Aug 04 10:27:39.949951 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:39.949924 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-113.ec2.internal\" not found" Aug 04 10:27:39.953200 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.953185 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3d393a7cd5ccc56f50081d0bd7ed54a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-113.ec2.internal\" (UID: \"e3d393a7cd5ccc56f50081d0bd7ed54a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-113.ec2.internal" Aug 04 10:27:39.953256 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.953209 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3d393a7cd5ccc56f50081d0bd7ed54a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-113.ec2.internal\" (UID: \"e3d393a7cd5ccc56f50081d0bd7ed54a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-113.ec2.internal" Aug 04 10:27:39.953256 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.953226 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1a9bc83f737210ec581d9649412c17cd-config\") pod \"kube-apiserver-proxy-ip-10-0-142-113.ec2.internal\" (UID: \"1a9bc83f737210ec581d9649412c17cd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-113.ec2.internal" Aug 04 10:27:39.953326 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.953267 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1a9bc83f737210ec581d9649412c17cd-config\") pod \"kube-apiserver-proxy-ip-10-0-142-113.ec2.internal\" (UID: \"1a9bc83f737210ec581d9649412c17cd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-113.ec2.internal" Aug 04 10:27:39.953326 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.953293 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3d393a7cd5ccc56f50081d0bd7ed54a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-113.ec2.internal\" (UID: \"e3d393a7cd5ccc56f50081d0bd7ed54a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-113.ec2.internal" Aug 04 10:27:39.953326 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:39.953297 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3d393a7cd5ccc56f50081d0bd7ed54a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-113.ec2.internal\" (UID: \"e3d393a7cd5ccc56f50081d0bd7ed54a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-113.ec2.internal" Aug 04 10:27:40.050658 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:40.050577 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-113.ec2.internal\" not found" Aug 04 10:27:40.132013 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:40.131969 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-113.ec2.internal" Aug 04 10:27:40.136876 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:40.136855 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-113.ec2.internal" Aug 04 10:27:41.544513 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:41.544480 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-113.ec2.internal\" not found" Aug 04 10:27:41.644927 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:41.644879 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-113.ec2.internal\" not found" Aug 04 10:27:41.745422 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:41.745347 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-113.ec2.internal\" not found" Aug 04 10:27:41.845810 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:41.845781 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-113.ec2.internal\" not found" Aug 04 10:27:41.884138 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:41.884105 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Aug 04 10:27:41.884604 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:41.884291 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 04 10:27:41.946332 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:41.946301 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-113.ec2.internal\" not found" Aug 04 10:27:41.960193 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:41.960152 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-03 10:22:40 +0000 UTC" deadline="2028-01-24 07:56:40.756180716 +0000 UTC" Aug 04 10:27:41.960193 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:41.960193 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12909h28m58.795992013s" Aug 04 10:27:42.046861 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:42.046838 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-113.ec2.internal\" not found" Aug 04 10:27:42.047253 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:42.047229 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d393a7cd5ccc56f50081d0bd7ed54a.slice/crio-b7254e5e00d8a7f163e4b862ab320916abffb7166b23e81572c3c46116a3a988 WatchSource:0}: Error finding container b7254e5e00d8a7f163e4b862ab320916abffb7166b23e81572c3c46116a3a988: Status 404 returned error can't find the container with id b7254e5e00d8a7f163e4b862ab320916abffb7166b23e81572c3c46116a3a988 Aug 04 10:27:42.049084 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:42.049060 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a9bc83f737210ec581d9649412c17cd.slice/crio-50d00f3aa8776ba05d1a15d64016e6b6ac9fe05009e5b1c290b160d09c5c153b WatchSource:0}: Error finding container 50d00f3aa8776ba05d1a15d64016e6b6ac9fe05009e5b1c290b160d09c5c153b: Status 404 returned error can't find the container with id 50d00f3aa8776ba05d1a15d64016e6b6ac9fe05009e5b1c290b160d09c5c153b Aug 04 10:27:42.052371 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.052355 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 10:27:42.097042 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.096980 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-113.ec2.internal" event={"ID":"1a9bc83f737210ec581d9649412c17cd","Type":"ContainerStarted","Data":"50d00f3aa8776ba05d1a15d64016e6b6ac9fe05009e5b1c290b160d09c5c153b"} Aug 04 10:27:42.098086 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.098061 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-113.ec2.internal" event={"ID":"e3d393a7cd5ccc56f50081d0bd7ed54a","Type":"ContainerStarted","Data":"b7254e5e00d8a7f163e4b862ab320916abffb7166b23e81572c3c46116a3a988"} Aug 04 10:27:42.147479 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:42.147435 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-113.ec2.internal\" not found" Aug 04 10:27:42.150130 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.150114 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 10:27:42.244635 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.244598 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-113.ec2.internal" Aug 04 10:27:42.258667 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.258640 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 10:27:42.261041 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.261024 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-113.ec2.internal" Aug 04 10:27:42.270198 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.270173 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 10:27:42.270641 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.270625 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 10:27:42.273521 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.273503 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 10:27:42.921926 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.921894 2580 apiserver.go:52] "Watching apiserver" Aug 04 10:27:42.933878 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.933630 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Aug 04 10:27:42.934095 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.934068 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-113.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6n67b","openshift-dns/node-resolver-m4sfm","openshift-multus/multus-additional-cni-plugins-r66c8","openshift-multus/network-metrics-daemon-wc2fl","openshift-network-operator/iptables-alerter-7bhbp","openshift-ovn-kubernetes/ovnkube-node-tb4zk","kube-system/global-pull-secret-syncer-rzhxt","kube-system/konnectivity-agent-tgfd7","openshift-cluster-node-tuning-operator/tuned-hjpbq","openshift-image-registry/node-ca-bhq6d","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-113.ec2.internal","openshift-multus/multus-49x8b","openshift-network-diagnostics/network-check-target-vnz6r"] Aug 04 10:27:42.935889 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.935822 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:42.937444 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.937422 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-49x8b" Aug 04 10:27:42.938423 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.938403 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m4sfm" Aug 04 10:27:42.941586 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.940047 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Aug 04 10:27:42.941586 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.940280 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Aug 04 10:27:42.941586 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.940464 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Aug 04 10:27:42.941586 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.940673 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Aug 04 10:27:42.941586 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.940972 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cjz64\"" Aug 04 10:27:42.941586 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.941178 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Aug 04 10:27:42.941586 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.941391 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Aug 04 10:27:42.942458 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.942137 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Aug 04 10:27:42.942458 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.942411 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Aug 04 10:27:42.945644 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.945626 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Aug 04 10:27:42.946366 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.946347 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Aug 04 10:27:42.949904 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.949862 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Aug 04 10:27:42.950100 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.950075 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r66c8" Aug 04 10:27:42.950155 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.950115 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Aug 04 10:27:42.950367 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.950353 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zkpq5\"" Aug 04 10:27:42.950527 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.950506 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc2fl" Aug 04 10:27:42.950733 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.950529 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-tdbds\"" Aug 04 10:27:42.950733 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:42.950594 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc2fl" podUID="0db48901-2221-480e-97b0-cd6be997aaef" Aug 04 10:27:42.950733 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.950679 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7bhbp" Aug 04 10:27:42.950892 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.950737 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Aug 04 10:27:42.950892 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.950772 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:27:42.950892 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:42.950833 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhxt" podUID="b0a98e70-0e23-4348-9b6d-93b4081069df" Aug 04 10:27:42.951124 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.951107 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tgfd7" Aug 04 10:27:42.951189 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.951121 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6n67b" Aug 04 10:27:42.951242 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.951201 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:42.951242 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.951207 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnz6r" Aug 04 10:27:42.951417 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:42.951396 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnz6r" podUID="b8276ab4-da95-49d7-81ed-bbc401ac406a" Aug 04 10:27:42.951476 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.951443 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bhq6d" Aug 04 10:27:42.954768 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.954747 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ffgmx\"" Aug 04 10:27:42.954919 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.954904 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-64vqm\"" Aug 04 10:27:42.954981 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.954925 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Aug 04 10:27:42.955141 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.955123 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjvbm\" (UniqueName: \"kubernetes.io/projected/768a1a06-b330-417b-9d56-91e051f96ecb-kube-api-access-cjvbm\") pod \"aws-ebs-csi-driver-node-6n67b\" (UID: \"768a1a06-b330-417b-9d56-91e051f96ecb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6n67b" Aug 04 10:27:42.955203 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.955158 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13819f5d-21f5-43f5-88f6-8bd915c7e86f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r66c8\" (UID: \"13819f5d-21f5-43f5-88f6-8bd915c7e86f\") " pod="openshift-multus/multus-additional-cni-plugins-r66c8" Aug 04 10:27:42.955264 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.955235 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-run-ovn\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:42.955315 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.955260 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-cnibin\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:42.955315 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.955278 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0626797c-56b0-4bac-b1a3-4a7d41a6707f-cni-binary-copy\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:42.955315 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.955295 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-host-run-netns\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:42.955460 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.955319 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-host-run-multus-certs\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:42.955460 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.955344 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-etc-kubernetes\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:42.955460 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.955367 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4tpg\" (UniqueName: \"kubernetes.io/projected/b8276ab4-da95-49d7-81ed-bbc401ac406a-kube-api-access-g4tpg\") pod \"network-check-target-vnz6r\" (UID: \"b8276ab4-da95-49d7-81ed-bbc401ac406a\") " pod="openshift-network-diagnostics/network-check-target-vnz6r" Aug 04 10:27:42.955460 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.955392 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/768a1a06-b330-417b-9d56-91e051f96ecb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6n67b\" (UID: \"768a1a06-b330-417b-9d56-91e051f96ecb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6n67b" Aug 04 10:27:42.955460 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.955414 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv2zc\" (UniqueName: \"kubernetes.io/projected/59e9a581-bf1d-4594-8d5b-9dba10576132-kube-api-access-tv2zc\") pod \"node-resolver-m4sfm\" (UID: \"59e9a581-bf1d-4594-8d5b-9dba10576132\") " pod="openshift-dns/node-resolver-m4sfm" Aug 04 10:27:42.955760 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.955461 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13819f5d-21f5-43f5-88f6-8bd915c7e86f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r66c8\" (UID: \"13819f5d-21f5-43f5-88f6-8bd915c7e86f\") " pod="openshift-multus/multus-additional-cni-plugins-r66c8" Aug 04 10:27:42.955760 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.955492 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0a98e70-0e23-4348-9b6d-93b4081069df-original-pull-secret\") pod \"global-pull-secret-syncer-rzhxt\" (UID: \"b0a98e70-0e23-4348-9b6d-93b4081069df\") " pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:27:42.955760 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.955547 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/768a1a06-b330-417b-9d56-91e051f96ecb-device-dir\") pod \"aws-ebs-csi-driver-node-6n67b\" (UID: \"768a1a06-b330-417b-9d56-91e051f96ecb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6n67b" Aug 04 10:27:42.955760 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.955598 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b0a98e70-0e23-4348-9b6d-93b4081069df-kubelet-config\") pod \"global-pull-secret-syncer-rzhxt\" (UID: \"b0a98e70-0e23-4348-9b6d-93b4081069df\") " pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:27:42.955760 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.955621 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-run-systemd\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:42.955760 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.955642 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c5922147-cb42-4f62-9880-8deaa51a4b0f-env-overrides\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:42.955760 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.955666 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/768a1a06-b330-417b-9d56-91e051f96ecb-registration-dir\") pod \"aws-ebs-csi-driver-node-6n67b\" (UID: \"768a1a06-b330-417b-9d56-91e051f96ecb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6n67b" Aug 04 10:27:42.955760 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.955695 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13819f5d-21f5-43f5-88f6-8bd915c7e86f-system-cni-dir\") pod \"multus-additional-cni-plugins-r66c8\" (UID: \"13819f5d-21f5-43f5-88f6-8bd915c7e86f\") " pod="openshift-multus/multus-additional-cni-plugins-r66c8" Aug 04 10:27:42.955760 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.955715 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/13819f5d-21f5-43f5-88f6-8bd915c7e86f-os-release\") pod \"multus-additional-cni-plugins-r66c8\" (UID: \"13819f5d-21f5-43f5-88f6-8bd915c7e86f\") " pod="openshift-multus/multus-additional-cni-plugins-r66c8" Aug 04 10:27:42.955760 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.955731 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-lib-modules\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:42.955760 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.955746 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-var-lib-openvswitch\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:42.955760 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.955760 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-etc-openvswitch\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:42.956278 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.955774 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vnpm\" (UniqueName: \"kubernetes.io/projected/beb5524a-082b-4e73-a9c3-b4c011b96944-kube-api-access-7vnpm\") pod \"iptables-alerter-7bhbp\" (UID: \"beb5524a-082b-4e73-a9c3-b4c011b96944\") " pod="openshift-network-operator/iptables-alerter-7bhbp" Aug 04 10:27:42.956278 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.955798 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/143c5d48-59e9-404c-bbd0-d18dfc3fe400-etc-tuned\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:42.956278 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.955814 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-run-openvswitch\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:42.956278 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.955829 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-hostroot\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:42.956278 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.955895 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-host\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:42.956278 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.955918 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-host-kubelet\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:42.956278 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.955939 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-node-log\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:42.956278 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.955968 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13819f5d-21f5-43f5-88f6-8bd915c7e86f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r66c8\" (UID: \"13819f5d-21f5-43f5-88f6-8bd915c7e86f\") " pod="openshift-multus/multus-additional-cni-plugins-r66c8" Aug 04 10:27:42.956278 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956028 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/143c5d48-59e9-404c-bbd0-d18dfc3fe400-tmp\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:42.956278 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956060 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fm6k\" (UniqueName: \"kubernetes.io/projected/143c5d48-59e9-404c-bbd0-d18dfc3fe400-kube-api-access-6fm6k\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:42.956278 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956086 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-os-release\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:42.956278 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956105 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-host-var-lib-cni-multus\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:42.956278 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956128 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0db48901-2221-480e-97b0-cd6be997aaef-metrics-certs\") pod \"network-metrics-daemon-wc2fl\" (UID: \"0db48901-2221-480e-97b0-cd6be997aaef\") " pod="openshift-multus/network-metrics-daemon-wc2fl" Aug 04 10:27:42.956278 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956149 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-etc-sysctl-conf\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:42.956278 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956183 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/768a1a06-b330-417b-9d56-91e051f96ecb-etc-selinux\") pod \"aws-ebs-csi-driver-node-6n67b\" (UID: \"768a1a06-b330-417b-9d56-91e051f96ecb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6n67b" Aug 04 10:27:42.956278 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956207 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsc8b\" (UniqueName: \"kubernetes.io/projected/13819f5d-21f5-43f5-88f6-8bd915c7e86f-kube-api-access-fsc8b\") pod \"multus-additional-cni-plugins-r66c8\" (UID: \"13819f5d-21f5-43f5-88f6-8bd915c7e86f\") " pod="openshift-multus/multus-additional-cni-plugins-r66c8" Aug 04 10:27:42.956961 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956245 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-etc-sysconfig\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:42.956961 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956270 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-systemd-units\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:42.956961 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956293 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-host-cni-netd\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:42.956961 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956338 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-host-var-lib-cni-bin\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:42.956961 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956386 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-etc-systemd\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:42.956961 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956411 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/768a1a06-b330-417b-9d56-91e051f96ecb-sys-fs\") pod \"aws-ebs-csi-driver-node-6n67b\" (UID: \"768a1a06-b330-417b-9d56-91e051f96ecb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6n67b" Aug 04 10:27:42.956961 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956432 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/beb5524a-082b-4e73-a9c3-b4c011b96944-host-slash\") pod \"iptables-alerter-7bhbp\" (UID: \"beb5524a-082b-4e73-a9c3-b4c011b96944\") " pod="openshift-network-operator/iptables-alerter-7bhbp" Aug 04 10:27:42.956961 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956466 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-etc-modprobe-d\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:42.956961 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956488 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:42.956961 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956511 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-multus-conf-dir\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:42.956961 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956536 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4xvc\" (UniqueName: \"kubernetes.io/projected/0626797c-56b0-4bac-b1a3-4a7d41a6707f-kube-api-access-z4xvc\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:42.956961 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956578 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e3137682-eadf-4ec1-b891-3e4fd7493df5-agent-certs\") pod \"konnectivity-agent-tgfd7\" (UID: \"e3137682-eadf-4ec1-b891-3e4fd7493df5\") " pod="kube-system/konnectivity-agent-tgfd7" Aug 04 10:27:42.956961 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956601 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e3137682-eadf-4ec1-b891-3e4fd7493df5-konnectivity-ca\") pod \"konnectivity-agent-tgfd7\" (UID: \"e3137682-eadf-4ec1-b891-3e4fd7493df5\") " pod="kube-system/konnectivity-agent-tgfd7" Aug 04 10:27:42.956961 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956655 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59e9a581-bf1d-4594-8d5b-9dba10576132-hosts-file\") pod \"node-resolver-m4sfm\" (UID: \"59e9a581-bf1d-4594-8d5b-9dba10576132\") " pod="openshift-dns/node-resolver-m4sfm" Aug 04 10:27:42.956961 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956677 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13819f5d-21f5-43f5-88f6-8bd915c7e86f-cnibin\") pod \"multus-additional-cni-plugins-r66c8\" (UID: \"13819f5d-21f5-43f5-88f6-8bd915c7e86f\") " pod="openshift-multus/multus-additional-cni-plugins-r66c8" Aug 04 10:27:42.956961 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956700 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-etc-kubernetes\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:42.957691 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956722 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-etc-sysctl-d\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:42.957691 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956744 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnmcf\" (UniqueName: \"kubernetes.io/projected/53b096bc-7df7-4067-8c04-fd4d68746d12-kube-api-access-lnmcf\") pod \"node-ca-bhq6d\" (UID: \"53b096bc-7df7-4067-8c04-fd4d68746d12\") " pod="openshift-image-registry/node-ca-bhq6d" Aug 04 10:27:42.957691 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956767 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-host-run-netns\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:42.957691 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956789 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c5922147-cb42-4f62-9880-8deaa51a4b0f-ovnkube-script-lib\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:42.957691 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956810 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-host-run-k8s-cni-cncf-io\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:42.957691 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956835 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53b096bc-7df7-4067-8c04-fd4d68746d12-host\") pod \"node-ca-bhq6d\" (UID: \"53b096bc-7df7-4067-8c04-fd4d68746d12\") " pod="openshift-image-registry/node-ca-bhq6d" Aug 04 10:27:42.957691 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956857 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-host-slash\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:42.957691 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956902 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c5922147-cb42-4f62-9880-8deaa51a4b0f-ovnkube-config\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:42.957691 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956926 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7lx9\" (UniqueName: \"kubernetes.io/projected/c5922147-cb42-4f62-9880-8deaa51a4b0f-kube-api-access-x7lx9\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:42.957691 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956951 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lskwn\" (UniqueName: \"kubernetes.io/projected/0db48901-2221-480e-97b0-cd6be997aaef-kube-api-access-lskwn\") pod \"network-metrics-daemon-wc2fl\" (UID: \"0db48901-2221-480e-97b0-cd6be997aaef\") " pod="openshift-multus/network-metrics-daemon-wc2fl" Aug 04 10:27:42.957691 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.956976 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/beb5524a-082b-4e73-a9c3-b4c011b96944-iptables-alerter-script\") pod \"iptables-alerter-7bhbp\" (UID: \"beb5524a-082b-4e73-a9c3-b4c011b96944\") " pod="openshift-network-operator/iptables-alerter-7bhbp" Aug 04 10:27:42.957691 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.957001 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/53b096bc-7df7-4067-8c04-fd4d68746d12-serviceca\") pod \"node-ca-bhq6d\" (UID: \"53b096bc-7df7-4067-8c04-fd4d68746d12\") " pod="openshift-image-registry/node-ca-bhq6d" Aug 04 10:27:42.957691 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.957025 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b0a98e70-0e23-4348-9b6d-93b4081069df-dbus\") pod \"global-pull-secret-syncer-rzhxt\" (UID: \"b0a98e70-0e23-4348-9b6d-93b4081069df\") " pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:27:42.957691 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.957048 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-system-cni-dir\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:42.957691 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.957071 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-multus-socket-dir-parent\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:42.957691 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.957095 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-host-var-lib-kubelet\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:42.958413 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.957119 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-host-cni-bin\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:42.958413 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.957145 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-run\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:42.958413 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.957167 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-sys\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:42.958413 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.957188 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-var-lib-kubelet\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:42.958413 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.957213 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-log-socket\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:42.958413 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.957236 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-host-run-ovn-kubernetes\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:42.958413 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.957260 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c5922147-cb42-4f62-9880-8deaa51a4b0f-ovn-node-metrics-cert\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:42.958413 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.957282 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-multus-cni-dir\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:42.958413 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.957304 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/768a1a06-b330-417b-9d56-91e051f96ecb-socket-dir\") pod \"aws-ebs-csi-driver-node-6n67b\" (UID: \"768a1a06-b330-417b-9d56-91e051f96ecb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6n67b" Aug 04 10:27:42.958413 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.957327 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59e9a581-bf1d-4594-8d5b-9dba10576132-tmp-dir\") pod \"node-resolver-m4sfm\" (UID: \"59e9a581-bf1d-4594-8d5b-9dba10576132\") " pod="openshift-dns/node-resolver-m4sfm" Aug 04 10:27:42.958413 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.957351 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13819f5d-21f5-43f5-88f6-8bd915c7e86f-cni-binary-copy\") pod \"multus-additional-cni-plugins-r66c8\" (UID: \"13819f5d-21f5-43f5-88f6-8bd915c7e86f\") " pod="openshift-multus/multus-additional-cni-plugins-r66c8" Aug 04 10:27:42.958413 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.957373 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0626797c-56b0-4bac-b1a3-4a7d41a6707f-multus-daemon-config\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:42.958413 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.957839 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Aug 04 10:27:42.958413 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.958037 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Aug 04 10:27:42.958413 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.958087 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pbb8t\"" Aug 04 10:27:42.958413 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.958218 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Aug 04 10:27:42.959233 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.958440 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Aug 04 10:27:42.959233 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.958453 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Aug 04 10:27:42.959233 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.958844 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sw94l\"" Aug 04 10:27:42.959364 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.959327 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2d8gs\"" Aug 04 10:27:42.959442 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.959423 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-r7bpb\"" Aug 04 10:27:42.959492 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.959330 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Aug 04 10:27:42.959862 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.959701 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Aug 04 10:27:42.959862 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.959730 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Aug 04 10:27:42.960024 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.960000 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Aug 04 10:27:42.960084 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.960053 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Aug 04 10:27:42.960244 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.960224 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Aug 04 10:27:42.960304 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.960246 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Aug 04 10:27:42.960355 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.960005 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Aug 04 10:27:42.960462 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.960444 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Aug 04 10:27:42.968487 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.968461 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 10:27:42.987484 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.986718 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 10:27:42.989456 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.988649 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5rnn7" Aug 04 10:27:42.996014 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:42.995534 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5rnn7" Aug 04 10:27:43.045261 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.045233 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Aug 04 10:27:43.057829 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.057781 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-run\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:43.057829 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.057836 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-sys\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:43.058072 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.057898 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-var-lib-kubelet\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:43.058072 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.057989 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-var-lib-kubelet\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:43.058072 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.057985 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-sys\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:43.058072 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.058016 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-log-socket\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.058072 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.058056 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-host-run-ovn-kubernetes\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.058072 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.058065 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-log-socket\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.058389 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.058083 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c5922147-cb42-4f62-9880-8deaa51a4b0f-ovn-node-metrics-cert\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.058389 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.058112 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-multus-cni-dir\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.058389 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.058105 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-host-run-ovn-kubernetes\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.058389 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.058146 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/768a1a06-b330-417b-9d56-91e051f96ecb-socket-dir\") pod \"aws-ebs-csi-driver-node-6n67b\" (UID: \"768a1a06-b330-417b-9d56-91e051f96ecb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6n67b" Aug 04 10:27:43.058389 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.058170 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59e9a581-bf1d-4594-8d5b-9dba10576132-tmp-dir\") pod \"node-resolver-m4sfm\" (UID: \"59e9a581-bf1d-4594-8d5b-9dba10576132\") " pod="openshift-dns/node-resolver-m4sfm" Aug 04 10:27:43.058389 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.058210 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13819f5d-21f5-43f5-88f6-8bd915c7e86f-cni-binary-copy\") pod \"multus-additional-cni-plugins-r66c8\" (UID: \"13819f5d-21f5-43f5-88f6-8bd915c7e86f\") " pod="openshift-multus/multus-additional-cni-plugins-r66c8" Aug 04 10:27:43.058389 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.058234 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0626797c-56b0-4bac-b1a3-4a7d41a6707f-multus-daemon-config\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.058389 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.058270 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-multus-cni-dir\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.058389 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.058262 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjvbm\" (UniqueName: \"kubernetes.io/projected/768a1a06-b330-417b-9d56-91e051f96ecb-kube-api-access-cjvbm\") pod \"aws-ebs-csi-driver-node-6n67b\" (UID: \"768a1a06-b330-417b-9d56-91e051f96ecb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6n67b" Aug 04 10:27:43.058389 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.058303 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13819f5d-21f5-43f5-88f6-8bd915c7e86f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r66c8\" (UID: \"13819f5d-21f5-43f5-88f6-8bd915c7e86f\") " pod="openshift-multus/multus-additional-cni-plugins-r66c8" Aug 04 10:27:43.058389 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.058362 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-run-ovn\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.058389 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.058388 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-cnibin\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.059034 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.058411 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0626797c-56b0-4bac-b1a3-4a7d41a6707f-cni-binary-copy\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.059034 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.058450 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-host-run-netns\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.059034 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.058476 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-host-run-multus-certs\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.059034 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.058521 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-etc-kubernetes\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.059034 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.058547 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4tpg\" (UniqueName: \"kubernetes.io/projected/b8276ab4-da95-49d7-81ed-bbc401ac406a-kube-api-access-g4tpg\") pod \"network-check-target-vnz6r\" (UID: \"b8276ab4-da95-49d7-81ed-bbc401ac406a\") " pod="openshift-network-diagnostics/network-check-target-vnz6r" Aug 04 10:27:43.059034 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.058595 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/768a1a06-b330-417b-9d56-91e051f96ecb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6n67b\" (UID: \"768a1a06-b330-417b-9d56-91e051f96ecb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6n67b" Aug 04 10:27:43.059034 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.058619 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv2zc\" (UniqueName: \"kubernetes.io/projected/59e9a581-bf1d-4594-8d5b-9dba10576132-kube-api-access-tv2zc\") pod \"node-resolver-m4sfm\" (UID: \"59e9a581-bf1d-4594-8d5b-9dba10576132\") " pod="openshift-dns/node-resolver-m4sfm" Aug 04 10:27:43.059034 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.058658 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13819f5d-21f5-43f5-88f6-8bd915c7e86f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r66c8\" (UID: \"13819f5d-21f5-43f5-88f6-8bd915c7e86f\") " pod="openshift-multus/multus-additional-cni-plugins-r66c8" Aug 04 10:27:43.059034 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.058696 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0a98e70-0e23-4348-9b6d-93b4081069df-original-pull-secret\") pod \"global-pull-secret-syncer-rzhxt\" (UID: \"b0a98e70-0e23-4348-9b6d-93b4081069df\") " pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:27:43.059034 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.058735 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/768a1a06-b330-417b-9d56-91e051f96ecb-device-dir\") pod \"aws-ebs-csi-driver-node-6n67b\" (UID: \"768a1a06-b330-417b-9d56-91e051f96ecb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6n67b" Aug 04 10:27:43.059034 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.058769 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b0a98e70-0e23-4348-9b6d-93b4081069df-kubelet-config\") pod \"global-pull-secret-syncer-rzhxt\" (UID: \"b0a98e70-0e23-4348-9b6d-93b4081069df\") " pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:27:43.059034 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.058794 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-run-systemd\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.059034 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.058851 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c5922147-cb42-4f62-9880-8deaa51a4b0f-env-overrides\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.059034 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.058900 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/768a1a06-b330-417b-9d56-91e051f96ecb-registration-dir\") pod \"aws-ebs-csi-driver-node-6n67b\" (UID: \"768a1a06-b330-417b-9d56-91e051f96ecb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6n67b" Aug 04 10:27:43.059034 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.058935 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13819f5d-21f5-43f5-88f6-8bd915c7e86f-system-cni-dir\") pod \"multus-additional-cni-plugins-r66c8\" (UID: \"13819f5d-21f5-43f5-88f6-8bd915c7e86f\") " pod="openshift-multus/multus-additional-cni-plugins-r66c8" Aug 04 10:27:43.059034 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.058978 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/13819f5d-21f5-43f5-88f6-8bd915c7e86f-os-release\") pod \"multus-additional-cni-plugins-r66c8\" (UID: \"13819f5d-21f5-43f5-88f6-8bd915c7e86f\") " pod="openshift-multus/multus-additional-cni-plugins-r66c8" Aug 04 10:27:43.059034 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059002 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-lib-modules\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:43.059967 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059026 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-var-lib-openvswitch\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.059967 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059077 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-etc-openvswitch\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.059967 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059106 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13819f5d-21f5-43f5-88f6-8bd915c7e86f-cni-binary-copy\") pod \"multus-additional-cni-plugins-r66c8\" (UID: \"13819f5d-21f5-43f5-88f6-8bd915c7e86f\") " pod="openshift-multus/multus-additional-cni-plugins-r66c8" Aug 04 10:27:43.059967 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059129 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vnpm\" (UniqueName: \"kubernetes.io/projected/beb5524a-082b-4e73-a9c3-b4c011b96944-kube-api-access-7vnpm\") pod \"iptables-alerter-7bhbp\" (UID: \"beb5524a-082b-4e73-a9c3-b4c011b96944\") " pod="openshift-network-operator/iptables-alerter-7bhbp" Aug 04 10:27:43.059967 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059164 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/143c5d48-59e9-404c-bbd0-d18dfc3fe400-etc-tuned\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:43.059967 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059203 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-run-openvswitch\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.059967 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059226 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-hostroot\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.059967 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059246 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/768a1a06-b330-417b-9d56-91e051f96ecb-socket-dir\") pod \"aws-ebs-csi-driver-node-6n67b\" (UID: \"768a1a06-b330-417b-9d56-91e051f96ecb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6n67b" Aug 04 10:27:43.059967 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059249 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-host\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:43.059967 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059298 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-host-kubelet\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.059967 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059326 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-node-log\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.059967 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059338 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-host\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:43.059967 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059362 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13819f5d-21f5-43f5-88f6-8bd915c7e86f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r66c8\" (UID: \"13819f5d-21f5-43f5-88f6-8bd915c7e86f\") " pod="openshift-multus/multus-additional-cni-plugins-r66c8" Aug 04 10:27:43.059967 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059398 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/143c5d48-59e9-404c-bbd0-d18dfc3fe400-tmp\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:43.059967 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059432 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fm6k\" (UniqueName: \"kubernetes.io/projected/143c5d48-59e9-404c-bbd0-d18dfc3fe400-kube-api-access-6fm6k\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:43.059967 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059464 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-os-release\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.059967 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059514 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-host-var-lib-cni-multus\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.060881 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059543 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0db48901-2221-480e-97b0-cd6be997aaef-metrics-certs\") pod \"network-metrics-daemon-wc2fl\" (UID: \"0db48901-2221-480e-97b0-cd6be997aaef\") " pod="openshift-multus/network-metrics-daemon-wc2fl" Aug 04 10:27:43.060881 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059596 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-etc-sysctl-conf\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:43.060881 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059625 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/768a1a06-b330-417b-9d56-91e051f96ecb-etc-selinux\") pod \"aws-ebs-csi-driver-node-6n67b\" (UID: \"768a1a06-b330-417b-9d56-91e051f96ecb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6n67b" Aug 04 10:27:43.060881 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059655 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsc8b\" (UniqueName: \"kubernetes.io/projected/13819f5d-21f5-43f5-88f6-8bd915c7e86f-kube-api-access-fsc8b\") pod \"multus-additional-cni-plugins-r66c8\" (UID: \"13819f5d-21f5-43f5-88f6-8bd915c7e86f\") " pod="openshift-multus/multus-additional-cni-plugins-r66c8" Aug 04 10:27:43.060881 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059684 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-etc-sysconfig\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:43.060881 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059712 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-systemd-units\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.060881 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059742 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-host-cni-netd\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.060881 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059770 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-host-var-lib-cni-bin\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.060881 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059799 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-etc-systemd\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:43.060881 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059828 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/768a1a06-b330-417b-9d56-91e051f96ecb-sys-fs\") pod \"aws-ebs-csi-driver-node-6n67b\" (UID: \"768a1a06-b330-417b-9d56-91e051f96ecb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6n67b" Aug 04 10:27:43.060881 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059858 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/beb5524a-082b-4e73-a9c3-b4c011b96944-host-slash\") pod \"iptables-alerter-7bhbp\" (UID: \"beb5524a-082b-4e73-a9c3-b4c011b96944\") " pod="openshift-network-operator/iptables-alerter-7bhbp" Aug 04 10:27:43.060881 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059888 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-etc-modprobe-d\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:43.060881 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059920 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.060881 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059956 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-multus-conf-dir\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.060881 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.059984 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4xvc\" (UniqueName: \"kubernetes.io/projected/0626797c-56b0-4bac-b1a3-4a7d41a6707f-kube-api-access-z4xvc\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.060881 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.060012 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e3137682-eadf-4ec1-b891-3e4fd7493df5-agent-certs\") pod \"konnectivity-agent-tgfd7\" (UID: \"e3137682-eadf-4ec1-b891-3e4fd7493df5\") " pod="kube-system/konnectivity-agent-tgfd7" Aug 04 10:27:43.060881 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.060043 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e3137682-eadf-4ec1-b891-3e4fd7493df5-konnectivity-ca\") pod \"konnectivity-agent-tgfd7\" (UID: \"e3137682-eadf-4ec1-b891-3e4fd7493df5\") " pod="kube-system/konnectivity-agent-tgfd7" Aug 04 10:27:43.061864 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.060071 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59e9a581-bf1d-4594-8d5b-9dba10576132-hosts-file\") pod \"node-resolver-m4sfm\" (UID: \"59e9a581-bf1d-4594-8d5b-9dba10576132\") " pod="openshift-dns/node-resolver-m4sfm" Aug 04 10:27:43.061864 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.060101 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13819f5d-21f5-43f5-88f6-8bd915c7e86f-cnibin\") pod \"multus-additional-cni-plugins-r66c8\" (UID: \"13819f5d-21f5-43f5-88f6-8bd915c7e86f\") " pod="openshift-multus/multus-additional-cni-plugins-r66c8" Aug 04 10:27:43.061864 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.060130 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-etc-kubernetes\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:43.061864 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.060157 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-etc-sysctl-d\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:43.061864 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.060186 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnmcf\" (UniqueName: \"kubernetes.io/projected/53b096bc-7df7-4067-8c04-fd4d68746d12-kube-api-access-lnmcf\") pod \"node-ca-bhq6d\" (UID: \"53b096bc-7df7-4067-8c04-fd4d68746d12\") " pod="openshift-image-registry/node-ca-bhq6d" Aug 04 10:27:43.061864 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.060216 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-host-run-netns\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.061864 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.060247 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c5922147-cb42-4f62-9880-8deaa51a4b0f-ovnkube-script-lib\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.061864 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.060276 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-host-run-k8s-cni-cncf-io\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.061864 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.060323 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53b096bc-7df7-4067-8c04-fd4d68746d12-host\") pod \"node-ca-bhq6d\" (UID: \"53b096bc-7df7-4067-8c04-fd4d68746d12\") " pod="openshift-image-registry/node-ca-bhq6d" Aug 04 10:27:43.061864 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.060326 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13819f5d-21f5-43f5-88f6-8bd915c7e86f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r66c8\" (UID: \"13819f5d-21f5-43f5-88f6-8bd915c7e86f\") " pod="openshift-multus/multus-additional-cni-plugins-r66c8" Aug 04 10:27:43.061864 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.060353 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-host-slash\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.061864 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.060381 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c5922147-cb42-4f62-9880-8deaa51a4b0f-ovnkube-config\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.061864 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.060412 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7lx9\" (UniqueName: \"kubernetes.io/projected/c5922147-cb42-4f62-9880-8deaa51a4b0f-kube-api-access-x7lx9\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.061864 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.060445 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lskwn\" (UniqueName: \"kubernetes.io/projected/0db48901-2221-480e-97b0-cd6be997aaef-kube-api-access-lskwn\") pod \"network-metrics-daemon-wc2fl\" (UID: \"0db48901-2221-480e-97b0-cd6be997aaef\") " pod="openshift-multus/network-metrics-daemon-wc2fl" Aug 04 10:27:43.061864 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.060477 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/beb5524a-082b-4e73-a9c3-b4c011b96944-iptables-alerter-script\") pod \"iptables-alerter-7bhbp\" (UID: \"beb5524a-082b-4e73-a9c3-b4c011b96944\") " pod="openshift-network-operator/iptables-alerter-7bhbp" Aug 04 10:27:43.061864 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.060526 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/53b096bc-7df7-4067-8c04-fd4d68746d12-serviceca\") pod \"node-ca-bhq6d\" (UID: \"53b096bc-7df7-4067-8c04-fd4d68746d12\") " pod="openshift-image-registry/node-ca-bhq6d" Aug 04 10:27:43.061864 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.060582 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b0a98e70-0e23-4348-9b6d-93b4081069df-dbus\") pod \"global-pull-secret-syncer-rzhxt\" (UID: \"b0a98e70-0e23-4348-9b6d-93b4081069df\") " pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:27:43.062815 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.060614 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-system-cni-dir\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.062815 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.060644 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-multus-socket-dir-parent\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.062815 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.060663 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Aug 04 10:27:43.062815 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.060678 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-host-var-lib-kubelet\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.062815 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.060707 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-host-cni-bin\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.062815 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.060910 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-host-cni-bin\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.062815 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.061320 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59e9a581-bf1d-4594-8d5b-9dba10576132-tmp-dir\") pod \"node-resolver-m4sfm\" (UID: \"59e9a581-bf1d-4594-8d5b-9dba10576132\") " pod="openshift-dns/node-resolver-m4sfm" Aug 04 10:27:43.062815 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.061414 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-host-kubelet\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.062815 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.061460 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-node-log\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.062815 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.061970 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13819f5d-21f5-43f5-88f6-8bd915c7e86f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r66c8\" (UID: \"13819f5d-21f5-43f5-88f6-8bd915c7e86f\") " pod="openshift-multus/multus-additional-cni-plugins-r66c8" Aug 04 10:27:43.062815 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.062508 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0626797c-56b0-4bac-b1a3-4a7d41a6707f-multus-daemon-config\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.062815 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.062640 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-run-ovn\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.062815 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.062702 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-cnibin\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.063502 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.063187 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0626797c-56b0-4bac-b1a3-4a7d41a6707f-cni-binary-copy\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.063502 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.063254 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-host-run-netns\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.063502 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.063305 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-host-run-multus-certs\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.063502 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.063354 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-etc-kubernetes\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.063742 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.063691 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/768a1a06-b330-417b-9d56-91e051f96ecb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6n67b\" (UID: \"768a1a06-b330-417b-9d56-91e051f96ecb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6n67b" Aug 04 10:27:43.066773 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.064148 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-host-cni-netd\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.066773 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.064369 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-os-release\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.066773 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.064431 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-host-var-lib-cni-multus\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.066773 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.064413 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e3137682-eadf-4ec1-b891-3e4fd7493df5-konnectivity-ca\") pod \"konnectivity-agent-tgfd7\" (UID: \"e3137682-eadf-4ec1-b891-3e4fd7493df5\") " pod="kube-system/konnectivity-agent-tgfd7" Aug 04 10:27:43.066773 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.064429 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13819f5d-21f5-43f5-88f6-8bd915c7e86f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r66c8\" (UID: \"13819f5d-21f5-43f5-88f6-8bd915c7e86f\") " pod="openshift-multus/multus-additional-cni-plugins-r66c8" Aug 04 10:27:43.066773 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.064507 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/768a1a06-b330-417b-9d56-91e051f96ecb-sys-fs\") pod \"aws-ebs-csi-driver-node-6n67b\" (UID: \"768a1a06-b330-417b-9d56-91e051f96ecb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6n67b" Aug 04 10:27:43.066773 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:43.064525 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 10:27:43.066773 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.064580 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-host-var-lib-cni-bin\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.066773 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.064601 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-etc-sysctl-conf\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:43.066773 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.064659 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-etc-systemd\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:43.066773 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.064680 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/768a1a06-b330-417b-9d56-91e051f96ecb-device-dir\") pod \"aws-ebs-csi-driver-node-6n67b\" (UID: \"768a1a06-b330-417b-9d56-91e051f96ecb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6n67b" Aug 04 10:27:43.066773 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:43.064722 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0a98e70-0e23-4348-9b6d-93b4081069df-original-pull-secret podName:b0a98e70-0e23-4348-9b6d-93b4081069df nodeName:}" failed. No retries permitted until 2026-08-04 10:27:43.564688255 +0000 UTC m=+2.985575659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0a98e70-0e23-4348-9b6d-93b4081069df-original-pull-secret") pod "global-pull-secret-syncer-rzhxt" (UID: "b0a98e70-0e23-4348-9b6d-93b4081069df") : object "kube-system"/"original-pull-secret" not registered Aug 04 10:27:43.066773 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.064737 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-run-systemd\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.066773 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.064773 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/beb5524a-082b-4e73-a9c3-b4c011b96944-host-slash\") pod \"iptables-alerter-7bhbp\" (UID: \"beb5524a-082b-4e73-a9c3-b4c011b96944\") " pod="openshift-network-operator/iptables-alerter-7bhbp" Aug 04 10:27:43.066773 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.064852 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c5922147-cb42-4f62-9880-8deaa51a4b0f-ovn-node-metrics-cert\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.066773 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.064884 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-etc-modprobe-d\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:43.066773 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.064891 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-etc-sysconfig\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:43.069151 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.064937 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.069151 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.064962 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-run\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:43.069151 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.065000 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-systemd-units\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.069151 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.065020 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-lib-modules\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:43.069151 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.065028 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/768a1a06-b330-417b-9d56-91e051f96ecb-etc-selinux\") pod \"aws-ebs-csi-driver-node-6n67b\" (UID: \"768a1a06-b330-417b-9d56-91e051f96ecb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6n67b" Aug 04 10:27:43.069151 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.065073 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b0a98e70-0e23-4348-9b6d-93b4081069df-kubelet-config\") pod \"global-pull-secret-syncer-rzhxt\" (UID: \"b0a98e70-0e23-4348-9b6d-93b4081069df\") " pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:27:43.069151 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:43.065078 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:27:43.069151 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.065107 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c5922147-cb42-4f62-9880-8deaa51a4b0f-env-overrides\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.069151 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.065109 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/768a1a06-b330-417b-9d56-91e051f96ecb-registration-dir\") pod \"aws-ebs-csi-driver-node-6n67b\" (UID: \"768a1a06-b330-417b-9d56-91e051f96ecb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6n67b" Aug 04 10:27:43.069151 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:43.065144 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0db48901-2221-480e-97b0-cd6be997aaef-metrics-certs podName:0db48901-2221-480e-97b0-cd6be997aaef nodeName:}" failed. No retries permitted until 2026-08-04 10:27:43.56512692 +0000 UTC m=+2.986014332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0db48901-2221-480e-97b0-cd6be997aaef-metrics-certs") pod "network-metrics-daemon-wc2fl" (UID: "0db48901-2221-480e-97b0-cd6be997aaef") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:27:43.069151 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.065141 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-multus-conf-dir\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.069151 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.065169 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-etc-openvswitch\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.069151 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.065172 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-var-lib-openvswitch\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.069151 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.065220 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-run-openvswitch\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.069151 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.065277 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-hostroot\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.069151 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.065506 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59e9a581-bf1d-4594-8d5b-9dba10576132-hosts-file\") pod \"node-resolver-m4sfm\" (UID: \"59e9a581-bf1d-4594-8d5b-9dba10576132\") " pod="openshift-dns/node-resolver-m4sfm" Aug 04 10:27:43.069151 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.065787 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-etc-kubernetes\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:43.070403 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.065848 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/13819f5d-21f5-43f5-88f6-8bd915c7e86f-os-release\") pod \"multus-additional-cni-plugins-r66c8\" (UID: \"13819f5d-21f5-43f5-88f6-8bd915c7e86f\") " pod="openshift-multus/multus-additional-cni-plugins-r66c8" Aug 04 10:27:43.070403 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.066074 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-host-run-k8s-cni-cncf-io\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.070403 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.066127 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13819f5d-21f5-43f5-88f6-8bd915c7e86f-cnibin\") pod \"multus-additional-cni-plugins-r66c8\" (UID: \"13819f5d-21f5-43f5-88f6-8bd915c7e86f\") " pod="openshift-multus/multus-additional-cni-plugins-r66c8" Aug 04 10:27:43.070403 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.066184 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53b096bc-7df7-4067-8c04-fd4d68746d12-host\") pod \"node-ca-bhq6d\" (UID: \"53b096bc-7df7-4067-8c04-fd4d68746d12\") " pod="openshift-image-registry/node-ca-bhq6d" Aug 04 10:27:43.070403 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.066327 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13819f5d-21f5-43f5-88f6-8bd915c7e86f-system-cni-dir\") pod \"multus-additional-cni-plugins-r66c8\" (UID: \"13819f5d-21f5-43f5-88f6-8bd915c7e86f\") " pod="openshift-multus/multus-additional-cni-plugins-r66c8" Aug 04 10:27:43.070403 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.066594 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-system-cni-dir\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.070403 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.066643 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c5922147-cb42-4f62-9880-8deaa51a4b0f-ovnkube-config\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.070403 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.066670 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b0a98e70-0e23-4348-9b6d-93b4081069df-dbus\") pod \"global-pull-secret-syncer-rzhxt\" (UID: \"b0a98e70-0e23-4348-9b6d-93b4081069df\") " pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:27:43.070403 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.066856 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c5922147-cb42-4f62-9880-8deaa51a4b0f-ovnkube-script-lib\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.070403 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.066887 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-multus-socket-dir-parent\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.070403 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.066934 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-host-run-netns\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.070403 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.067033 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/143c5d48-59e9-404c-bbd0-d18dfc3fe400-etc-sysctl-d\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:43.070403 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.067071 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5922147-cb42-4f62-9880-8deaa51a4b0f-host-slash\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.070403 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.067316 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/beb5524a-082b-4e73-a9c3-b4c011b96944-iptables-alerter-script\") pod \"iptables-alerter-7bhbp\" (UID: \"beb5524a-082b-4e73-a9c3-b4c011b96944\") " pod="openshift-network-operator/iptables-alerter-7bhbp" Aug 04 10:27:43.070403 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.067365 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0626797c-56b0-4bac-b1a3-4a7d41a6707f-host-var-lib-kubelet\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.070403 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.066783 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/53b096bc-7df7-4067-8c04-fd4d68746d12-serviceca\") pod \"node-ca-bhq6d\" (UID: \"53b096bc-7df7-4067-8c04-fd4d68746d12\") " pod="openshift-image-registry/node-ca-bhq6d" Aug 04 10:27:43.070403 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.069622 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/143c5d48-59e9-404c-bbd0-d18dfc3fe400-etc-tuned\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:43.071451 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.071429 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e3137682-eadf-4ec1-b891-3e4fd7493df5-agent-certs\") pod \"konnectivity-agent-tgfd7\" (UID: \"e3137682-eadf-4ec1-b891-3e4fd7493df5\") " pod="kube-system/konnectivity-agent-tgfd7" Aug 04 10:27:43.071699 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.071649 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjvbm\" (UniqueName: \"kubernetes.io/projected/768a1a06-b330-417b-9d56-91e051f96ecb-kube-api-access-cjvbm\") pod \"aws-ebs-csi-driver-node-6n67b\" (UID: \"768a1a06-b330-417b-9d56-91e051f96ecb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6n67b" Aug 04 10:27:43.071797 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:43.071737 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 10:27:43.071969 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:43.071915 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 10:27:43.071969 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:43.071934 2580 projected.go:196] Error preparing data for projected volume kube-api-access-g4tpg for pod openshift-network-diagnostics/network-check-target-vnz6r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:27:43.072117 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:43.072039 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8276ab4-da95-49d7-81ed-bbc401ac406a-kube-api-access-g4tpg podName:b8276ab4-da95-49d7-81ed-bbc401ac406a nodeName:}" failed. No retries permitted until 2026-08-04 10:27:43.572017938 +0000 UTC m=+2.992905350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-g4tpg" (UniqueName: "kubernetes.io/projected/b8276ab4-da95-49d7-81ed-bbc401ac406a-kube-api-access-g4tpg") pod "network-check-target-vnz6r" (UID: "b8276ab4-da95-49d7-81ed-bbc401ac406a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:27:43.074039 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.074016 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/143c5d48-59e9-404c-bbd0-d18dfc3fe400-tmp\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:43.080938 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.080857 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsc8b\" (UniqueName: \"kubernetes.io/projected/13819f5d-21f5-43f5-88f6-8bd915c7e86f-kube-api-access-fsc8b\") pod \"multus-additional-cni-plugins-r66c8\" (UID: \"13819f5d-21f5-43f5-88f6-8bd915c7e86f\") " pod="openshift-multus/multus-additional-cni-plugins-r66c8" Aug 04 10:27:43.083792 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.083767 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7lx9\" (UniqueName: \"kubernetes.io/projected/c5922147-cb42-4f62-9880-8deaa51a4b0f-kube-api-access-x7lx9\") pod \"ovnkube-node-tb4zk\" (UID: \"c5922147-cb42-4f62-9880-8deaa51a4b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.083908 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.083811 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4xvc\" (UniqueName: \"kubernetes.io/projected/0626797c-56b0-4bac-b1a3-4a7d41a6707f-kube-api-access-z4xvc\") pod \"multus-49x8b\" (UID: \"0626797c-56b0-4bac-b1a3-4a7d41a6707f\") " pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.084093 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.084072 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv2zc\" (UniqueName: \"kubernetes.io/projected/59e9a581-bf1d-4594-8d5b-9dba10576132-kube-api-access-tv2zc\") pod \"node-resolver-m4sfm\" (UID: \"59e9a581-bf1d-4594-8d5b-9dba10576132\") " pod="openshift-dns/node-resolver-m4sfm" Aug 04 10:27:43.084229 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.084185 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fm6k\" (UniqueName: \"kubernetes.io/projected/143c5d48-59e9-404c-bbd0-d18dfc3fe400-kube-api-access-6fm6k\") pod \"tuned-hjpbq\" (UID: \"143c5d48-59e9-404c-bbd0-d18dfc3fe400\") " pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:43.084663 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.084613 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnmcf\" (UniqueName: \"kubernetes.io/projected/53b096bc-7df7-4067-8c04-fd4d68746d12-kube-api-access-lnmcf\") pod \"node-ca-bhq6d\" (UID: \"53b096bc-7df7-4067-8c04-fd4d68746d12\") " pod="openshift-image-registry/node-ca-bhq6d" Aug 04 10:27:43.085627 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.085609 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lskwn\" (UniqueName: \"kubernetes.io/projected/0db48901-2221-480e-97b0-cd6be997aaef-kube-api-access-lskwn\") pod \"network-metrics-daemon-wc2fl\" (UID: \"0db48901-2221-480e-97b0-cd6be997aaef\") " pod="openshift-multus/network-metrics-daemon-wc2fl" Aug 04 10:27:43.086030 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.086013 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vnpm\" (UniqueName: \"kubernetes.io/projected/beb5524a-082b-4e73-a9c3-b4c011b96944-kube-api-access-7vnpm\") pod \"iptables-alerter-7bhbp\" (UID: \"beb5524a-082b-4e73-a9c3-b4c011b96944\") " pod="openshift-network-operator/iptables-alerter-7bhbp" Aug 04 10:27:43.269197 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.269103 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:27:43.287825 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.287071 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-49x8b" Aug 04 10:27:43.290929 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.290903 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m4sfm" Aug 04 10:27:43.299150 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.299123 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r66c8" Aug 04 10:27:43.307969 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.307945 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7bhbp" Aug 04 10:27:43.317751 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.317727 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tgfd7" Aug 04 10:27:43.327320 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.327296 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" Aug 04 10:27:43.337965 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.337937 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6n67b" Aug 04 10:27:43.345662 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.345639 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bhq6d" Aug 04 10:27:43.664548 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.664516 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4tpg\" (UniqueName: \"kubernetes.io/projected/b8276ab4-da95-49d7-81ed-bbc401ac406a-kube-api-access-g4tpg\") pod \"network-check-target-vnz6r\" (UID: \"b8276ab4-da95-49d7-81ed-bbc401ac406a\") " pod="openshift-network-diagnostics/network-check-target-vnz6r" Aug 04 10:27:43.664757 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.664574 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0a98e70-0e23-4348-9b6d-93b4081069df-original-pull-secret\") pod \"global-pull-secret-syncer-rzhxt\" (UID: \"b0a98e70-0e23-4348-9b6d-93b4081069df\") " pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:27:43.664757 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.664612 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0db48901-2221-480e-97b0-cd6be997aaef-metrics-certs\") pod \"network-metrics-daemon-wc2fl\" (UID: \"0db48901-2221-480e-97b0-cd6be997aaef\") " pod="openshift-multus/network-metrics-daemon-wc2fl" Aug 04 10:27:43.664757 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:43.664708 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 10:27:43.664757 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:43.664718 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 10:27:43.664757 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:43.664735 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 10:27:43.664757 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:43.664748 2580 projected.go:196] Error preparing data for projected volume kube-api-access-g4tpg for pod openshift-network-diagnostics/network-check-target-vnz6r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:27:43.664966 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:43.664779 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0a98e70-0e23-4348-9b6d-93b4081069df-original-pull-secret podName:b0a98e70-0e23-4348-9b6d-93b4081069df nodeName:}" failed. No retries permitted until 2026-08-04 10:27:44.664764848 +0000 UTC m=+4.085652262 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0a98e70-0e23-4348-9b6d-93b4081069df-original-pull-secret") pod "global-pull-secret-syncer-rzhxt" (UID: "b0a98e70-0e23-4348-9b6d-93b4081069df") : object "kube-system"/"original-pull-secret" not registered Aug 04 10:27:43.664966 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:43.664718 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:27:43.664966 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:43.664794 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8276ab4-da95-49d7-81ed-bbc401ac406a-kube-api-access-g4tpg podName:b8276ab4-da95-49d7-81ed-bbc401ac406a nodeName:}" failed. No retries permitted until 2026-08-04 10:27:44.664788189 +0000 UTC m=+4.085675577 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-g4tpg" (UniqueName: "kubernetes.io/projected/b8276ab4-da95-49d7-81ed-bbc401ac406a-kube-api-access-g4tpg") pod "network-check-target-vnz6r" (UID: "b8276ab4-da95-49d7-81ed-bbc401ac406a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:27:43.664966 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:43.664818 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0db48901-2221-480e-97b0-cd6be997aaef-metrics-certs podName:0db48901-2221-480e-97b0-cd6be997aaef nodeName:}" failed. No retries permitted until 2026-08-04 10:27:44.664800954 +0000 UTC m=+4.085688347 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0db48901-2221-480e-97b0-cd6be997aaef-metrics-certs") pod "network-metrics-daemon-wc2fl" (UID: "0db48901-2221-480e-97b0-cd6be997aaef") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:27:43.968728 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:43.968631 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13819f5d_21f5_43f5_88f6_8bd915c7e86f.slice/crio-a6c45156253238a88b3038e294eb4ec2d1c08f759185aa101823ddf78944461d WatchSource:0}: Error finding container a6c45156253238a88b3038e294eb4ec2d1c08f759185aa101823ddf78944461d: Status 404 returned error can't find the container with id a6c45156253238a88b3038e294eb4ec2d1c08f759185aa101823ddf78944461d Aug 04 10:27:43.991541 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:43.991503 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0626797c_56b0_4bac_b1a3_4a7d41a6707f.slice/crio-4b4386b2dddafe3d161ea17c5be151091dea79cb789064165f82f3fe3a019b36 WatchSource:0}: Error finding container 4b4386b2dddafe3d161ea17c5be151091dea79cb789064165f82f3fe3a019b36: Status 404 returned error can't find the container with id 4b4386b2dddafe3d161ea17c5be151091dea79cb789064165f82f3fe3a019b36 Aug 04 10:27:43.993064 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:43.993038 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5922147_cb42_4f62_9880_8deaa51a4b0f.slice/crio-d2c469d3497122339af785363d5d0b71343279f94d02e313fc2cf0e443fc28ac WatchSource:0}: Error finding container d2c469d3497122339af785363d5d0b71343279f94d02e313fc2cf0e443fc28ac: Status 404 returned error can't find the container with id d2c469d3497122339af785363d5d0b71343279f94d02e313fc2cf0e443fc28ac Aug 04 10:27:43.994303 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:43.994276 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59e9a581_bf1d_4594_8d5b_9dba10576132.slice/crio-cafb05340c53fb1c8740f96eae84d6c14c07b55b4e542f542c689911d8c4d83f WatchSource:0}: Error finding container cafb05340c53fb1c8740f96eae84d6c14c07b55b4e542f542c689911d8c4d83f: Status 404 returned error can't find the container with id cafb05340c53fb1c8740f96eae84d6c14c07b55b4e542f542c689911d8c4d83f Aug 04 10:27:43.997004 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:43.995973 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb5524a_082b_4e73_a9c3_b4c011b96944.slice/crio-9d09c7bb85a2ebc7fefd99061130aa0c35b0caf71260a9686ff95d6e349a71f9 WatchSource:0}: Error finding container 9d09c7bb85a2ebc7fefd99061130aa0c35b0caf71260a9686ff95d6e349a71f9: Status 404 returned error can't find the container with id 9d09c7bb85a2ebc7fefd99061130aa0c35b0caf71260a9686ff95d6e349a71f9 Aug 04 10:27:43.997693 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.997483 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 10:22:42 +0000 UTC" deadline="2028-03-07 19:04:14.840306287 +0000 UTC" Aug 04 10:27:43.997693 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:43.997519 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13952h36m30.842791488s" Aug 04 10:27:43.999140 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:43.999085 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53b096bc_7df7_4067_8c04_fd4d68746d12.slice/crio-8bdeceea877a52940f749eaa75e59e21599c9e5334051cba76c98334e20b0471 WatchSource:0}: Error finding container 8bdeceea877a52940f749eaa75e59e21599c9e5334051cba76c98334e20b0471: Status 404 returned error can't find the container with id 8bdeceea877a52940f749eaa75e59e21599c9e5334051cba76c98334e20b0471 Aug 04 10:27:44.000347 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:44.000325 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod143c5d48_59e9_404c_bbd0_d18dfc3fe400.slice/crio-d3f56fa2307a1aa975ee59d811cf2993bbe218d754acf47ef7b6ba70098e36f3 WatchSource:0}: Error finding container d3f56fa2307a1aa975ee59d811cf2993bbe218d754acf47ef7b6ba70098e36f3: Status 404 returned error can't find the container with id d3f56fa2307a1aa975ee59d811cf2993bbe218d754acf47ef7b6ba70098e36f3 Aug 04 10:27:44.001212 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:44.001118 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3137682_eadf_4ec1_b891_3e4fd7493df5.slice/crio-bbaaa87ea256c71ad842cc76fd5b9590c23136eaae1b866a8680e36892159606 WatchSource:0}: Error finding container bbaaa87ea256c71ad842cc76fd5b9590c23136eaae1b866a8680e36892159606: Status 404 returned error can't find the container with id bbaaa87ea256c71ad842cc76fd5b9590c23136eaae1b866a8680e36892159606 Aug 04 10:27:44.002133 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:27:44.002049 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod768a1a06_b330_417b_9d56_91e051f96ecb.slice/crio-bf852bccebe63f3004672f297a7648252c459bb135178473fa932e31d62d01d9 WatchSource:0}: Error finding container bf852bccebe63f3004672f297a7648252c459bb135178473fa932e31d62d01d9: Status 404 returned error can't find the container with id bf852bccebe63f3004672f297a7648252c459bb135178473fa932e31d62d01d9 Aug 04 10:27:44.093918 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:44.093750 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:27:44.094049 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:44.094002 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhxt" podUID="b0a98e70-0e23-4348-9b6d-93b4081069df" Aug 04 10:27:44.104153 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:44.104120 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-49x8b" event={"ID":"0626797c-56b0-4bac-b1a3-4a7d41a6707f","Type":"ContainerStarted","Data":"4b4386b2dddafe3d161ea17c5be151091dea79cb789064165f82f3fe3a019b36"} Aug 04 10:27:44.106830 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:44.106804 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" event={"ID":"143c5d48-59e9-404c-bbd0-d18dfc3fe400","Type":"ContainerStarted","Data":"d3f56fa2307a1aa975ee59d811cf2993bbe218d754acf47ef7b6ba70098e36f3"} Aug 04 10:27:44.108743 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:44.108717 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7bhbp" event={"ID":"beb5524a-082b-4e73-a9c3-b4c011b96944","Type":"ContainerStarted","Data":"9d09c7bb85a2ebc7fefd99061130aa0c35b0caf71260a9686ff95d6e349a71f9"} Aug 04 10:27:44.110052 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:44.110033 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r66c8" event={"ID":"13819f5d-21f5-43f5-88f6-8bd915c7e86f","Type":"ContainerStarted","Data":"a6c45156253238a88b3038e294eb4ec2d1c08f759185aa101823ddf78944461d"} Aug 04 10:27:44.113360 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:44.113338 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-113.ec2.internal" event={"ID":"1a9bc83f737210ec581d9649412c17cd","Type":"ContainerStarted","Data":"6fb9cde2ab9767c10310d74980b0246de2e97359e07a71b4709feab6c8627a38"} Aug 04 10:27:44.115360 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:44.115340 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tgfd7" event={"ID":"e3137682-eadf-4ec1-b891-3e4fd7493df5","Type":"ContainerStarted","Data":"bbaaa87ea256c71ad842cc76fd5b9590c23136eaae1b866a8680e36892159606"} Aug 04 10:27:44.118187 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:44.118157 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6n67b" event={"ID":"768a1a06-b330-417b-9d56-91e051f96ecb","Type":"ContainerStarted","Data":"bf852bccebe63f3004672f297a7648252c459bb135178473fa932e31d62d01d9"} Aug 04 10:27:44.119807 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:44.119786 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bhq6d" event={"ID":"53b096bc-7df7-4067-8c04-fd4d68746d12","Type":"ContainerStarted","Data":"8bdeceea877a52940f749eaa75e59e21599c9e5334051cba76c98334e20b0471"} Aug 04 10:27:44.122089 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:44.122037 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m4sfm" event={"ID":"59e9a581-bf1d-4594-8d5b-9dba10576132","Type":"ContainerStarted","Data":"cafb05340c53fb1c8740f96eae84d6c14c07b55b4e542f542c689911d8c4d83f"} Aug 04 10:27:44.123005 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:44.122986 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" event={"ID":"c5922147-cb42-4f62-9880-8deaa51a4b0f","Type":"ContainerStarted","Data":"d2c469d3497122339af785363d5d0b71343279f94d02e313fc2cf0e443fc28ac"} Aug 04 10:27:44.130874 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:44.130832 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-113.ec2.internal" podStartSLOduration=2.130819695 podStartE2EDuration="2.130819695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:27:44.130615306 +0000 UTC m=+3.551502722" watchObservedRunningTime="2026-08-04 10:27:44.130819695 +0000 UTC m=+3.551707105" Aug 04 10:27:44.671463 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:44.671420 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0db48901-2221-480e-97b0-cd6be997aaef-metrics-certs\") pod \"network-metrics-daemon-wc2fl\" (UID: \"0db48901-2221-480e-97b0-cd6be997aaef\") " pod="openshift-multus/network-metrics-daemon-wc2fl" Aug 04 10:27:44.671704 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:44.671518 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4tpg\" (UniqueName: \"kubernetes.io/projected/b8276ab4-da95-49d7-81ed-bbc401ac406a-kube-api-access-g4tpg\") pod \"network-check-target-vnz6r\" (UID: \"b8276ab4-da95-49d7-81ed-bbc401ac406a\") " pod="openshift-network-diagnostics/network-check-target-vnz6r" Aug 04 10:27:44.671704 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:44.671549 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0a98e70-0e23-4348-9b6d-93b4081069df-original-pull-secret\") pod \"global-pull-secret-syncer-rzhxt\" (UID: \"b0a98e70-0e23-4348-9b6d-93b4081069df\") " pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:27:44.671832 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:44.671710 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 10:27:44.671832 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:44.671778 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0a98e70-0e23-4348-9b6d-93b4081069df-original-pull-secret podName:b0a98e70-0e23-4348-9b6d-93b4081069df nodeName:}" failed. No retries permitted until 2026-08-04 10:27:46.671758989 +0000 UTC m=+6.092646383 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0a98e70-0e23-4348-9b6d-93b4081069df-original-pull-secret") pod "global-pull-secret-syncer-rzhxt" (UID: "b0a98e70-0e23-4348-9b6d-93b4081069df") : object "kube-system"/"original-pull-secret" not registered Aug 04 10:27:44.671945 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:44.671856 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:27:44.671945 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:44.671894 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0db48901-2221-480e-97b0-cd6be997aaef-metrics-certs podName:0db48901-2221-480e-97b0-cd6be997aaef nodeName:}" failed. No retries permitted until 2026-08-04 10:27:46.671881239 +0000 UTC m=+6.092768633 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0db48901-2221-480e-97b0-cd6be997aaef-metrics-certs") pod "network-metrics-daemon-wc2fl" (UID: "0db48901-2221-480e-97b0-cd6be997aaef") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:27:44.672050 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:44.671987 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 10:27:44.672050 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:44.672001 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 10:27:44.672050 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:44.672015 2580 projected.go:196] Error preparing data for projected volume kube-api-access-g4tpg for pod openshift-network-diagnostics/network-check-target-vnz6r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:27:44.672193 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:44.672053 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8276ab4-da95-49d7-81ed-bbc401ac406a-kube-api-access-g4tpg podName:b8276ab4-da95-49d7-81ed-bbc401ac406a nodeName:}" failed. No retries permitted until 2026-08-04 10:27:46.672041319 +0000 UTC m=+6.092928720 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-g4tpg" (UniqueName: "kubernetes.io/projected/b8276ab4-da95-49d7-81ed-bbc401ac406a-kube-api-access-g4tpg") pod "network-check-target-vnz6r" (UID: "b8276ab4-da95-49d7-81ed-bbc401ac406a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:27:44.998026 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:44.997909 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 10:22:42 +0000 UTC" deadline="2028-03-10 09:54:37.299054254 +0000 UTC" Aug 04 10:27:44.998026 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:44.997987 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14015h26m52.301080415s" Aug 04 10:27:45.095334 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:45.094683 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnz6r" Aug 04 10:27:45.095334 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:45.094714 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc2fl" Aug 04 10:27:45.095334 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:45.094814 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnz6r" podUID="b8276ab4-da95-49d7-81ed-bbc401ac406a" Aug 04 10:27:45.095334 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:45.095290 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc2fl" podUID="0db48901-2221-480e-97b0-cd6be997aaef" Aug 04 10:27:45.157355 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:45.156191 2580 generic.go:356] "Generic (PLEG): container finished" podID="e3d393a7cd5ccc56f50081d0bd7ed54a" containerID="f69c536d01aa2a3c56e5d42ae6c162c3491bc00f1b1363ee38f1e357c8abd758" exitCode=0 Aug 04 10:27:45.157355 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:45.157124 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-113.ec2.internal" event={"ID":"e3d393a7cd5ccc56f50081d0bd7ed54a","Type":"ContainerDied","Data":"f69c536d01aa2a3c56e5d42ae6c162c3491bc00f1b1363ee38f1e357c8abd758"} Aug 04 10:27:46.093791 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:46.093738 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:27:46.094393 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:46.093884 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhxt" podUID="b0a98e70-0e23-4348-9b6d-93b4081069df" Aug 04 10:27:46.175937 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:46.175892 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-113.ec2.internal" event={"ID":"e3d393a7cd5ccc56f50081d0bd7ed54a","Type":"ContainerStarted","Data":"ac9c4bc081e81226ec6db6d4e854e4ac42260ab6495ca853935c987f18e58467"} Aug 04 10:27:46.197076 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:46.196662 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-113.ec2.internal" podStartSLOduration=4.196639768 podStartE2EDuration="4.196639768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:27:46.196613728 +0000 UTC m=+5.617501137" watchObservedRunningTime="2026-08-04 10:27:46.196639768 +0000 UTC m=+5.617527181" Aug 04 10:27:46.690904 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:46.690860 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4tpg\" (UniqueName: \"kubernetes.io/projected/b8276ab4-da95-49d7-81ed-bbc401ac406a-kube-api-access-g4tpg\") pod \"network-check-target-vnz6r\" (UID: \"b8276ab4-da95-49d7-81ed-bbc401ac406a\") " pod="openshift-network-diagnostics/network-check-target-vnz6r" Aug 04 10:27:46.691108 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:46.690916 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0a98e70-0e23-4348-9b6d-93b4081069df-original-pull-secret\") pod \"global-pull-secret-syncer-rzhxt\" (UID: \"b0a98e70-0e23-4348-9b6d-93b4081069df\") " pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:27:46.691108 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:46.690954 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0db48901-2221-480e-97b0-cd6be997aaef-metrics-certs\") pod \"network-metrics-daemon-wc2fl\" (UID: \"0db48901-2221-480e-97b0-cd6be997aaef\") " pod="openshift-multus/network-metrics-daemon-wc2fl" Aug 04 10:27:46.691108 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:46.691044 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:27:46.691108 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:46.691045 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 10:27:46.691108 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:46.691069 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 10:27:46.691108 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:46.691088 2580 projected.go:196] Error preparing data for projected volume kube-api-access-g4tpg for pod openshift-network-diagnostics/network-check-target-vnz6r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:27:46.691108 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:46.691106 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0db48901-2221-480e-97b0-cd6be997aaef-metrics-certs podName:0db48901-2221-480e-97b0-cd6be997aaef nodeName:}" failed. No retries permitted until 2026-08-04 10:27:50.691086768 +0000 UTC m=+10.111974162 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0db48901-2221-480e-97b0-cd6be997aaef-metrics-certs") pod "network-metrics-daemon-wc2fl" (UID: "0db48901-2221-480e-97b0-cd6be997aaef") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:27:46.691476 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:46.691122 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8276ab4-da95-49d7-81ed-bbc401ac406a-kube-api-access-g4tpg podName:b8276ab4-da95-49d7-81ed-bbc401ac406a nodeName:}" failed. No retries permitted until 2026-08-04 10:27:50.691112294 +0000 UTC m=+10.111999697 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-g4tpg" (UniqueName: "kubernetes.io/projected/b8276ab4-da95-49d7-81ed-bbc401ac406a-kube-api-access-g4tpg") pod "network-check-target-vnz6r" (UID: "b8276ab4-da95-49d7-81ed-bbc401ac406a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:27:46.691476 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:46.691173 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 10:27:46.691476 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:46.691202 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0a98e70-0e23-4348-9b6d-93b4081069df-original-pull-secret podName:b0a98e70-0e23-4348-9b6d-93b4081069df nodeName:}" failed. No retries permitted until 2026-08-04 10:27:50.691192548 +0000 UTC m=+10.112079938 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0a98e70-0e23-4348-9b6d-93b4081069df-original-pull-secret") pod "global-pull-secret-syncer-rzhxt" (UID: "b0a98e70-0e23-4348-9b6d-93b4081069df") : object "kube-system"/"original-pull-secret" not registered Aug 04 10:27:47.096538 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:47.096461 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc2fl" Aug 04 10:27:47.096974 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:47.096587 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc2fl" podUID="0db48901-2221-480e-97b0-cd6be997aaef" Aug 04 10:27:47.096974 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:47.096926 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnz6r" Aug 04 10:27:47.097086 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:47.096994 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnz6r" podUID="b8276ab4-da95-49d7-81ed-bbc401ac406a" Aug 04 10:27:48.093983 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:48.093942 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:27:48.094189 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:48.094098 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhxt" podUID="b0a98e70-0e23-4348-9b6d-93b4081069df" Aug 04 10:27:49.096333 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:49.095705 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc2fl" Aug 04 10:27:49.096333 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:49.095841 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc2fl" podUID="0db48901-2221-480e-97b0-cd6be997aaef" Aug 04 10:27:49.096333 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:49.096201 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnz6r" Aug 04 10:27:49.096333 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:49.096284 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnz6r" podUID="b8276ab4-da95-49d7-81ed-bbc401ac406a" Aug 04 10:27:50.094457 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:50.093981 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:27:50.094457 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:50.094121 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhxt" podUID="b0a98e70-0e23-4348-9b6d-93b4081069df" Aug 04 10:27:50.727612 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:50.727574 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4tpg\" (UniqueName: \"kubernetes.io/projected/b8276ab4-da95-49d7-81ed-bbc401ac406a-kube-api-access-g4tpg\") pod \"network-check-target-vnz6r\" (UID: \"b8276ab4-da95-49d7-81ed-bbc401ac406a\") " pod="openshift-network-diagnostics/network-check-target-vnz6r" Aug 04 10:27:50.727987 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:50.727629 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0a98e70-0e23-4348-9b6d-93b4081069df-original-pull-secret\") pod \"global-pull-secret-syncer-rzhxt\" (UID: \"b0a98e70-0e23-4348-9b6d-93b4081069df\") " pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:27:50.727987 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:50.727667 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0db48901-2221-480e-97b0-cd6be997aaef-metrics-certs\") pod \"network-metrics-daemon-wc2fl\" (UID: \"0db48901-2221-480e-97b0-cd6be997aaef\") " pod="openshift-multus/network-metrics-daemon-wc2fl" Aug 04 10:27:50.727987 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:50.727744 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 10:27:50.727987 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:50.727765 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 10:27:50.727987 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:50.727778 2580 projected.go:196] Error preparing data for projected volume kube-api-access-g4tpg for pod openshift-network-diagnostics/network-check-target-vnz6r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:27:50.727987 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:50.727792 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 10:27:50.727987 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:50.727834 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0a98e70-0e23-4348-9b6d-93b4081069df-original-pull-secret podName:b0a98e70-0e23-4348-9b6d-93b4081069df nodeName:}" failed. No retries permitted until 2026-08-04 10:27:58.727817264 +0000 UTC m=+18.148704668 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0a98e70-0e23-4348-9b6d-93b4081069df-original-pull-secret") pod "global-pull-secret-syncer-rzhxt" (UID: "b0a98e70-0e23-4348-9b6d-93b4081069df") : object "kube-system"/"original-pull-secret" not registered Aug 04 10:27:50.727987 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:50.727850 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8276ab4-da95-49d7-81ed-bbc401ac406a-kube-api-access-g4tpg podName:b8276ab4-da95-49d7-81ed-bbc401ac406a nodeName:}" failed. No retries permitted until 2026-08-04 10:27:58.727840755 +0000 UTC m=+18.148728143 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-g4tpg" (UniqueName: "kubernetes.io/projected/b8276ab4-da95-49d7-81ed-bbc401ac406a-kube-api-access-g4tpg") pod "network-check-target-vnz6r" (UID: "b8276ab4-da95-49d7-81ed-bbc401ac406a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:27:50.727987 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:50.727749 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:27:50.727987 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:50.727878 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0db48901-2221-480e-97b0-cd6be997aaef-metrics-certs podName:0db48901-2221-480e-97b0-cd6be997aaef nodeName:}" failed. No retries permitted until 2026-08-04 10:27:58.727870086 +0000 UTC m=+18.148757475 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0db48901-2221-480e-97b0-cd6be997aaef-metrics-certs") pod "network-metrics-daemon-wc2fl" (UID: "0db48901-2221-480e-97b0-cd6be997aaef") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:27:51.095517 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:51.095022 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnz6r" Aug 04 10:27:51.095517 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:51.095129 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc2fl" Aug 04 10:27:51.095517 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:51.095260 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc2fl" podUID="0db48901-2221-480e-97b0-cd6be997aaef" Aug 04 10:27:51.095517 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:51.095134 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnz6r" podUID="b8276ab4-da95-49d7-81ed-bbc401ac406a" Aug 04 10:27:52.093836 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:52.093801 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:27:52.094313 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:52.093926 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhxt" podUID="b0a98e70-0e23-4348-9b6d-93b4081069df" Aug 04 10:27:53.095078 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:53.094552 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc2fl" Aug 04 10:27:53.095078 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:53.094715 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc2fl" podUID="0db48901-2221-480e-97b0-cd6be997aaef" Aug 04 10:27:53.095800 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:53.095652 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnz6r" Aug 04 10:27:53.095800 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:53.095767 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnz6r" podUID="b8276ab4-da95-49d7-81ed-bbc401ac406a" Aug 04 10:27:54.094164 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:54.094134 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:27:54.094315 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:54.094244 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhxt" podUID="b0a98e70-0e23-4348-9b6d-93b4081069df" Aug 04 10:27:55.093335 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:55.093300 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc2fl" Aug 04 10:27:55.093789 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:55.093343 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnz6r" Aug 04 10:27:55.093789 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:55.093463 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc2fl" podUID="0db48901-2221-480e-97b0-cd6be997aaef" Aug 04 10:27:55.093789 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:55.093595 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnz6r" podUID="b8276ab4-da95-49d7-81ed-bbc401ac406a" Aug 04 10:27:56.094246 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:56.094209 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:27:56.094688 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:56.094344 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhxt" podUID="b0a98e70-0e23-4348-9b6d-93b4081069df" Aug 04 10:27:57.094111 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:57.094071 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc2fl" Aug 04 10:27:57.094319 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:57.094129 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnz6r" Aug 04 10:27:57.094319 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:57.094226 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc2fl" podUID="0db48901-2221-480e-97b0-cd6be997aaef" Aug 04 10:27:57.094726 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:57.094358 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnz6r" podUID="b8276ab4-da95-49d7-81ed-bbc401ac406a" Aug 04 10:27:58.093947 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:58.093910 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:27:58.094140 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:58.094034 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhxt" podUID="b0a98e70-0e23-4348-9b6d-93b4081069df" Aug 04 10:27:58.793142 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:58.793092 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0db48901-2221-480e-97b0-cd6be997aaef-metrics-certs\") pod \"network-metrics-daemon-wc2fl\" (UID: \"0db48901-2221-480e-97b0-cd6be997aaef\") " pod="openshift-multus/network-metrics-daemon-wc2fl" Aug 04 10:27:58.793603 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:58.793170 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4tpg\" (UniqueName: \"kubernetes.io/projected/b8276ab4-da95-49d7-81ed-bbc401ac406a-kube-api-access-g4tpg\") pod \"network-check-target-vnz6r\" (UID: \"b8276ab4-da95-49d7-81ed-bbc401ac406a\") " pod="openshift-network-diagnostics/network-check-target-vnz6r" Aug 04 10:27:58.793603 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:58.793191 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0a98e70-0e23-4348-9b6d-93b4081069df-original-pull-secret\") pod \"global-pull-secret-syncer-rzhxt\" (UID: \"b0a98e70-0e23-4348-9b6d-93b4081069df\") " pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:27:58.793603 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:58.793306 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 10:27:58.793603 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:58.793369 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0a98e70-0e23-4348-9b6d-93b4081069df-original-pull-secret podName:b0a98e70-0e23-4348-9b6d-93b4081069df nodeName:}" failed. No retries permitted until 2026-08-04 10:28:14.793354763 +0000 UTC m=+34.214242156 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0a98e70-0e23-4348-9b6d-93b4081069df-original-pull-secret") pod "global-pull-secret-syncer-rzhxt" (UID: "b0a98e70-0e23-4348-9b6d-93b4081069df") : object "kube-system"/"original-pull-secret" not registered Aug 04 10:27:58.793603 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:58.793435 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:27:58.793603 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:58.793467 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0db48901-2221-480e-97b0-cd6be997aaef-metrics-certs podName:0db48901-2221-480e-97b0-cd6be997aaef nodeName:}" failed. No retries permitted until 2026-08-04 10:28:14.793459434 +0000 UTC m=+34.214346823 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0db48901-2221-480e-97b0-cd6be997aaef-metrics-certs") pod "network-metrics-daemon-wc2fl" (UID: "0db48901-2221-480e-97b0-cd6be997aaef") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:27:58.793603 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:58.793532 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 10:27:58.793603 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:58.793544 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 10:27:58.793603 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:58.793555 2580 projected.go:196] Error preparing data for projected volume kube-api-access-g4tpg for pod openshift-network-diagnostics/network-check-target-vnz6r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:27:58.793603 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:58.793608 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8276ab4-da95-49d7-81ed-bbc401ac406a-kube-api-access-g4tpg podName:b8276ab4-da95-49d7-81ed-bbc401ac406a nodeName:}" failed. No retries permitted until 2026-08-04 10:28:14.7935969 +0000 UTC m=+34.214484310 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-g4tpg" (UniqueName: "kubernetes.io/projected/b8276ab4-da95-49d7-81ed-bbc401ac406a-kube-api-access-g4tpg") pod "network-check-target-vnz6r" (UID: "b8276ab4-da95-49d7-81ed-bbc401ac406a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:27:59.093620 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:59.093587 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnz6r" Aug 04 10:27:59.093807 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:59.093712 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnz6r" podUID="b8276ab4-da95-49d7-81ed-bbc401ac406a" Aug 04 10:27:59.093807 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:27:59.093776 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc2fl" Aug 04 10:27:59.093918 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:27:59.093901 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc2fl" podUID="0db48901-2221-480e-97b0-cd6be997aaef" Aug 04 10:28:00.094005 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:00.093967 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:28:00.094401 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:00.094091 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhxt" podUID="b0a98e70-0e23-4348-9b6d-93b4081069df" Aug 04 10:28:01.094431 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:01.094398 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc2fl" Aug 04 10:28:01.094872 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:01.094541 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc2fl" podUID="0db48901-2221-480e-97b0-cd6be997aaef" Aug 04 10:28:01.094872 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:01.094581 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnz6r" Aug 04 10:28:01.094872 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:01.094671 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnz6r" podUID="b8276ab4-da95-49d7-81ed-bbc401ac406a" Aug 04 10:28:02.093949 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:02.093913 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:28:02.094146 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:02.094033 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhxt" podUID="b0a98e70-0e23-4348-9b6d-93b4081069df" Aug 04 10:28:03.094383 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:03.094060 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc2fl" Aug 04 10:28:03.095420 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:03.094104 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnz6r" Aug 04 10:28:03.095420 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:03.094495 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc2fl" podUID="0db48901-2221-480e-97b0-cd6be997aaef" Aug 04 10:28:03.095420 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:03.094520 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnz6r" podUID="b8276ab4-da95-49d7-81ed-bbc401ac406a" Aug 04 10:28:03.213326 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:03.213211 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tgfd7" event={"ID":"e3137682-eadf-4ec1-b891-3e4fd7493df5","Type":"ContainerStarted","Data":"27630118b8a852b11519f85ad8de91d9d7d53c2982c05de5f2a82c45bbd2aa2c"} Aug 04 10:28:03.214778 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:03.214752 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6n67b" event={"ID":"768a1a06-b330-417b-9d56-91e051f96ecb","Type":"ContainerStarted","Data":"84e755d1f328d2d4bd4d9ef11daf30c662df03dc9fec5d20f5a3e598f40f9853"} Aug 04 10:28:03.216161 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:03.216136 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bhq6d" event={"ID":"53b096bc-7df7-4067-8c04-fd4d68746d12","Type":"ContainerStarted","Data":"cc7293127c3c95cf8c69c153f85aec8c65a580d0504a4a14fd022d7c10afbfc6"} Aug 04 10:28:03.217680 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:03.217654 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m4sfm" event={"ID":"59e9a581-bf1d-4594-8d5b-9dba10576132","Type":"ContainerStarted","Data":"b835e93b1654547b66c839fab9912b119a3a4dac9b95eca0b8be76f5092f9593"} Aug 04 10:28:03.220686 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:03.220663 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tb4zk_c5922147-cb42-4f62-9880-8deaa51a4b0f/ovn-acl-logging/0.log" Aug 04 10:28:03.221040 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:03.221021 2580 generic.go:356] "Generic (PLEG): container finished" podID="c5922147-cb42-4f62-9880-8deaa51a4b0f" containerID="6570a33a86273637765ecee6d68765af2914449cc5d153726547a025dbe60785" exitCode=1 Aug 04 10:28:03.221115 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:03.221092 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" event={"ID":"c5922147-cb42-4f62-9880-8deaa51a4b0f","Type":"ContainerStarted","Data":"8ef8ef7e0a8c228d033d0093a7132d3270230fbf92417da3df837ddb406bee6d"} Aug 04 10:28:03.221194 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:03.221179 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" event={"ID":"c5922147-cb42-4f62-9880-8deaa51a4b0f","Type":"ContainerStarted","Data":"cc2d8da29d91def00e73318c6d63263b1d7c3bacafd2fb879941a6ca60ec0aff"} Aug 04 10:28:03.221250 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:03.221232 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" event={"ID":"c5922147-cb42-4f62-9880-8deaa51a4b0f","Type":"ContainerStarted","Data":"2594c30858d48ad2f7e3bb81faba22bf7e8b9b5f68393f741cfab5d976386815"} Aug 04 10:28:03.221301 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:03.221284 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" event={"ID":"c5922147-cb42-4f62-9880-8deaa51a4b0f","Type":"ContainerStarted","Data":"687ccf4de7d6be6cb34ca8f608dc301d0ab07b15494e2619d7d42c3ff31d855d"} Aug 04 10:28:03.221364 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:03.221342 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" event={"ID":"c5922147-cb42-4f62-9880-8deaa51a4b0f","Type":"ContainerDied","Data":"6570a33a86273637765ecee6d68765af2914449cc5d153726547a025dbe60785"} Aug 04 10:28:03.221419 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:03.221404 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" event={"ID":"c5922147-cb42-4f62-9880-8deaa51a4b0f","Type":"ContainerStarted","Data":"ea7a07b5a5329cebcd32418a3d696406afecb276207665eeba725b668a2906c0"} Aug 04 10:28:03.222525 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:03.222504 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-49x8b" event={"ID":"0626797c-56b0-4bac-b1a3-4a7d41a6707f","Type":"ContainerStarted","Data":"0f105f865bc3b9de9e1c865097c2e18d1127881cf796606b0f9f85c5c081c508"} Aug 04 10:28:03.224032 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:03.224004 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" event={"ID":"143c5d48-59e9-404c-bbd0-d18dfc3fe400","Type":"ContainerStarted","Data":"b1fc52180857a1a82d6b9b77b97cf5929bf4c3f335de471663a6df7bba54ea9a"} Aug 04 10:28:03.228177 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:03.228050 2580 generic.go:356] "Generic (PLEG): container finished" podID="13819f5d-21f5-43f5-88f6-8bd915c7e86f" containerID="a2fd659b0aabcee97883050fec619ce1f1d7c352c0447c018132944179301315" exitCode=0 Aug 04 10:28:03.228177 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:03.228092 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r66c8" event={"ID":"13819f5d-21f5-43f5-88f6-8bd915c7e86f","Type":"ContainerDied","Data":"a2fd659b0aabcee97883050fec619ce1f1d7c352c0447c018132944179301315"} Aug 04 10:28:03.233521 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:03.233482 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tgfd7" podStartSLOduration=3.889321329 podStartE2EDuration="22.233468886s" totalImagesPullingTime="18.344147557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:27:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:03.233417702 +0000 UTC m=+22.654305116" watchObservedRunningTime="2026-08-04 10:28:03.233468886 +0000 UTC m=+22.654356297" Aug 04 10:28:03.260201 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:03.260149 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hjpbq" podStartSLOduration=3.8672203290000002 podStartE2EDuration="22.260129518s" totalImagesPullingTime="18.392909189s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:27:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:03.25961838 +0000 UTC m=+22.680505783" watchObservedRunningTime="2026-08-04 10:28:03.260129518 +0000 UTC m=+22.681016929" Aug 04 10:28:03.277155 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:03.277106 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bhq6d" podStartSLOduration=3.885321664 podStartE2EDuration="22.277083768s" totalImagesPullingTime="18.391762104s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:27:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:03.276955495 +0000 UTC m=+22.697842929" watchObservedRunningTime="2026-08-04 10:28:03.277083768 +0000 UTC m=+22.697971179" Aug 04 10:28:03.297761 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:03.297716 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-49x8b" podStartSLOduration=3.7387788459999998 podStartE2EDuration="22.297697203s" totalImagesPullingTime="18.558918357s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:27:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:03.297358333 +0000 UTC m=+22.718245733" watchObservedRunningTime="2026-08-04 10:28:03.297697203 +0000 UTC m=+22.718584616" Aug 04 10:28:03.325527 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:03.325467 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-m4sfm" podStartSLOduration=3.929029345 podStartE2EDuration="22.325445529s" totalImagesPullingTime="18.396416184s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:27:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:03.324670174 +0000 UTC m=+22.745557586" watchObservedRunningTime="2026-08-04 10:28:03.325445529 +0000 UTC m=+22.746332940" Aug 04 10:28:04.093583 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:04.093535 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:28:04.093783 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:04.093748 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhxt" podUID="b0a98e70-0e23-4348-9b6d-93b4081069df" Aug 04 10:28:04.177240 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:04.177211 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Aug 04 10:28:04.232539 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:04.232503 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6n67b" event={"ID":"768a1a06-b330-417b-9d56-91e051f96ecb","Type":"ContainerStarted","Data":"22e7489ef31ec7dd8e2abe261fb7d605d61a60db72f01ecd3b42b459f77a39c9"} Aug 04 10:28:04.233991 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:04.233962 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7bhbp" event={"ID":"beb5524a-082b-4e73-a9c3-b4c011b96944","Type":"ContainerStarted","Data":"98e1064671a9cb5d98d8208b2cb72139738ba503b5b528d65497a5df623fadfa"} Aug 04 10:28:04.251810 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:04.251763 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7bhbp" podStartSLOduration=4.850987435 podStartE2EDuration="23.251747365s" totalImagesPullingTime="18.40075993s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:27:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:04.251421984 +0000 UTC m=+23.672309394" watchObservedRunningTime="2026-08-04 10:28:04.251747365 +0000 UTC m=+23.672634776" Aug 04 10:28:05.008582 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:05.008463 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-04T10:28:04.177231732Z","UUID":"8b6e6f67-cfaa-4a25-8085-14d2a92b6049","Handler":null,"Name":"","Endpoint":""} Aug 04 10:28:05.010828 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:05.010285 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Aug 04 10:28:05.010828 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:05.010333 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Aug 04 10:28:05.094109 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:05.094073 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc2fl" Aug 04 10:28:05.094288 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:05.094120 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnz6r" Aug 04 10:28:05.094288 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:05.094228 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc2fl" podUID="0db48901-2221-480e-97b0-cd6be997aaef" Aug 04 10:28:05.094402 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:05.094359 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnz6r" podUID="b8276ab4-da95-49d7-81ed-bbc401ac406a" Aug 04 10:28:05.372173 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:05.372139 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tgfd7" Aug 04 10:28:05.373101 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:05.373078 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tgfd7" Aug 04 10:28:06.094083 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:06.093922 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:28:06.094251 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:06.094152 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhxt" podUID="b0a98e70-0e23-4348-9b6d-93b4081069df" Aug 04 10:28:06.241519 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:06.241493 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tb4zk_c5922147-cb42-4f62-9880-8deaa51a4b0f/ovn-acl-logging/0.log" Aug 04 10:28:06.241896 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:06.241876 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" event={"ID":"c5922147-cb42-4f62-9880-8deaa51a4b0f","Type":"ContainerStarted","Data":"cca6892af28f60fc44547298a7006064798503e7d18c8a92fe6a937db472886c"} Aug 04 10:28:06.243747 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:06.243719 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6n67b" event={"ID":"768a1a06-b330-417b-9d56-91e051f96ecb","Type":"ContainerStarted","Data":"3d99015440374fb6c55b9e0c6df6b920aac45f9d292b320933ac0ea46325c8b5"} Aug 04 10:28:06.244136 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:06.244120 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tgfd7" Aug 04 10:28:06.244545 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:06.244531 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tgfd7" Aug 04 10:28:06.267484 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:06.267435 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6n67b" podStartSLOduration=4.124664937 podStartE2EDuration="25.267420488s" totalImagesPullingTime="21.142755551s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:27:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:06.267237186 +0000 UTC m=+25.688124609" watchObservedRunningTime="2026-08-04 10:28:06.267420488 +0000 UTC m=+25.688307898" Aug 04 10:28:07.094117 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:07.094079 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc2fl" Aug 04 10:28:07.094658 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:07.094130 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnz6r" Aug 04 10:28:07.094658 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:07.094238 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc2fl" podUID="0db48901-2221-480e-97b0-cd6be997aaef" Aug 04 10:28:07.094658 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:07.094329 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnz6r" podUID="b8276ab4-da95-49d7-81ed-bbc401ac406a" Aug 04 10:28:08.093308 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:08.093277 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:28:08.093498 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:08.093428 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhxt" podUID="b0a98e70-0e23-4348-9b6d-93b4081069df" Aug 04 10:28:09.094074 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:09.093791 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc2fl" Aug 04 10:28:09.094667 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:09.093843 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnz6r" Aug 04 10:28:09.094667 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:09.094139 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc2fl" podUID="0db48901-2221-480e-97b0-cd6be997aaef" Aug 04 10:28:09.094667 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:09.094167 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnz6r" podUID="b8276ab4-da95-49d7-81ed-bbc401ac406a" Aug 04 10:28:09.252660 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:09.252628 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tb4zk_c5922147-cb42-4f62-9880-8deaa51a4b0f/ovn-acl-logging/0.log" Aug 04 10:28:09.253006 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:09.252986 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" event={"ID":"c5922147-cb42-4f62-9880-8deaa51a4b0f","Type":"ContainerStarted","Data":"151287162d57f1af6441999bbabb2624fe6427f73b77261dcdbd7b64ba343e55"} Aug 04 10:28:09.253345 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:09.253306 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:28:09.253449 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:09.253384 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:28:09.253501 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:09.253482 2580 scope.go:117] "RemoveContainer" containerID="6570a33a86273637765ecee6d68765af2914449cc5d153726547a025dbe60785" Aug 04 10:28:09.254877 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:09.254857 2580 generic.go:356] "Generic (PLEG): container finished" podID="13819f5d-21f5-43f5-88f6-8bd915c7e86f" containerID="bdd4e823aa5822d88152aa5ad73ba2a7e81c359f130e86a867e0b3814c5632f7" exitCode=0 Aug 04 10:28:09.254959 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:09.254887 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r66c8" event={"ID":"13819f5d-21f5-43f5-88f6-8bd915c7e86f","Type":"ContainerDied","Data":"bdd4e823aa5822d88152aa5ad73ba2a7e81c359f130e86a867e0b3814c5632f7"} Aug 04 10:28:09.271239 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:09.271221 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:28:10.093514 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:10.093479 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:28:10.093660 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:10.093639 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhxt" podUID="b0a98e70-0e23-4348-9b6d-93b4081069df" Aug 04 10:28:10.260918 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:10.260765 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tb4zk_c5922147-cb42-4f62-9880-8deaa51a4b0f/ovn-acl-logging/0.log" Aug 04 10:28:10.261400 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:10.261366 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" event={"ID":"c5922147-cb42-4f62-9880-8deaa51a4b0f","Type":"ContainerStarted","Data":"36cde521ed882559020468bc3d424f8ad1e19aa036835243a6f82cbbf48608f3"} Aug 04 10:28:10.261839 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:10.261734 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:28:10.264187 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:10.264151 2580 generic.go:356] "Generic (PLEG): container finished" podID="13819f5d-21f5-43f5-88f6-8bd915c7e86f" containerID="74cfdf0b9ac5384bdfaa73d656527bdd191d6280f4bf3d54bffab3c3c3d6ea26" exitCode=0 Aug 04 10:28:10.264187 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:10.264186 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r66c8" event={"ID":"13819f5d-21f5-43f5-88f6-8bd915c7e86f","Type":"ContainerDied","Data":"74cfdf0b9ac5384bdfaa73d656527bdd191d6280f4bf3d54bffab3c3c3d6ea26"} Aug 04 10:28:10.280133 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:10.280107 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:28:10.297660 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:10.297616 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" podStartSLOduration=10.869824094 podStartE2EDuration="29.29758232s" totalImagesPullingTime="18.427758226s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:27:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:10.296067965 +0000 UTC m=+29.716955376" watchObservedRunningTime="2026-08-04 10:28:10.29758232 +0000 UTC m=+29.718469722" Aug 04 10:28:10.446576 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:10.446518 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rzhxt"] Aug 04 10:28:10.446745 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:10.446670 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:28:10.446817 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:10.446778 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhxt" podUID="b0a98e70-0e23-4348-9b6d-93b4081069df" Aug 04 10:28:10.449828 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:10.449799 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vnz6r"] Aug 04 10:28:10.449947 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:10.449920 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnz6r" Aug 04 10:28:10.450035 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:10.450013 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnz6r" podUID="b8276ab4-da95-49d7-81ed-bbc401ac406a" Aug 04 10:28:10.450314 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:10.450285 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wc2fl"] Aug 04 10:28:10.450409 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:10.450388 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc2fl" Aug 04 10:28:10.450484 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:10.450466 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc2fl" podUID="0db48901-2221-480e-97b0-cd6be997aaef" Aug 04 10:28:11.268774 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:11.268738 2580 generic.go:356] "Generic (PLEG): container finished" podID="13819f5d-21f5-43f5-88f6-8bd915c7e86f" containerID="8a1e460a0fab42c133626120fa77ce2ba7d69306e0b700e86e48cac3e3efedcd" exitCode=0 Aug 04 10:28:11.269402 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:11.268817 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r66c8" event={"ID":"13819f5d-21f5-43f5-88f6-8bd915c7e86f","Type":"ContainerDied","Data":"8a1e460a0fab42c133626120fa77ce2ba7d69306e0b700e86e48cac3e3efedcd"} Aug 04 10:28:12.094326 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:12.094291 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:28:12.094479 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:12.094290 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc2fl" Aug 04 10:28:12.094479 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:12.094439 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhxt" podUID="b0a98e70-0e23-4348-9b6d-93b4081069df" Aug 04 10:28:12.094596 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:12.094542 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc2fl" podUID="0db48901-2221-480e-97b0-cd6be997aaef" Aug 04 10:28:12.094596 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:12.094291 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnz6r" Aug 04 10:28:12.094708 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:12.094655 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnz6r" podUID="b8276ab4-da95-49d7-81ed-bbc401ac406a" Aug 04 10:28:14.093448 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:14.093409 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnz6r" Aug 04 10:28:14.093448 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:14.093440 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc2fl" Aug 04 10:28:14.094116 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:14.093431 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:28:14.094116 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:14.093545 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnz6r" podUID="b8276ab4-da95-49d7-81ed-bbc401ac406a" Aug 04 10:28:14.094116 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:14.093636 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhxt" podUID="b0a98e70-0e23-4348-9b6d-93b4081069df" Aug 04 10:28:14.094116 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:14.093712 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc2fl" podUID="0db48901-2221-480e-97b0-cd6be997aaef" Aug 04 10:28:14.814353 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:14.814306 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0db48901-2221-480e-97b0-cd6be997aaef-metrics-certs\") pod \"network-metrics-daemon-wc2fl\" (UID: \"0db48901-2221-480e-97b0-cd6be997aaef\") " pod="openshift-multus/network-metrics-daemon-wc2fl" Aug 04 10:28:14.814587 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:14.814393 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4tpg\" (UniqueName: \"kubernetes.io/projected/b8276ab4-da95-49d7-81ed-bbc401ac406a-kube-api-access-g4tpg\") pod \"network-check-target-vnz6r\" (UID: \"b8276ab4-da95-49d7-81ed-bbc401ac406a\") " pod="openshift-network-diagnostics/network-check-target-vnz6r" Aug 04 10:28:14.814587 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:14.814418 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0a98e70-0e23-4348-9b6d-93b4081069df-original-pull-secret\") pod \"global-pull-secret-syncer-rzhxt\" (UID: \"b0a98e70-0e23-4348-9b6d-93b4081069df\") " pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:28:14.814587 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:14.814451 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:28:14.814587 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:14.814507 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 10:28:14.814587 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:14.814528 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0db48901-2221-480e-97b0-cd6be997aaef-metrics-certs podName:0db48901-2221-480e-97b0-cd6be997aaef nodeName:}" failed. No retries permitted until 2026-08-04 10:28:46.814508688 +0000 UTC m=+66.235396098 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0db48901-2221-480e-97b0-cd6be997aaef-metrics-certs") pod "network-metrics-daemon-wc2fl" (UID: "0db48901-2221-480e-97b0-cd6be997aaef") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:28:14.814587 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:14.814555 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0a98e70-0e23-4348-9b6d-93b4081069df-original-pull-secret podName:b0a98e70-0e23-4348-9b6d-93b4081069df nodeName:}" failed. No retries permitted until 2026-08-04 10:28:46.814543882 +0000 UTC m=+66.235431283 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0a98e70-0e23-4348-9b6d-93b4081069df-original-pull-secret") pod "global-pull-secret-syncer-rzhxt" (UID: "b0a98e70-0e23-4348-9b6d-93b4081069df") : object "kube-system"/"original-pull-secret" not registered Aug 04 10:28:14.814587 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:14.814576 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 10:28:14.814970 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:14.814603 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 10:28:14.814970 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:14.814617 2580 projected.go:196] Error preparing data for projected volume kube-api-access-g4tpg for pod openshift-network-diagnostics/network-check-target-vnz6r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:28:14.814970 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:14.814663 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8276ab4-da95-49d7-81ed-bbc401ac406a-kube-api-access-g4tpg podName:b8276ab4-da95-49d7-81ed-bbc401ac406a nodeName:}" failed. No retries permitted until 2026-08-04 10:28:46.81465246 +0000 UTC m=+66.235539868 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-g4tpg" (UniqueName: "kubernetes.io/projected/b8276ab4-da95-49d7-81ed-bbc401ac406a-kube-api-access-g4tpg") pod "network-check-target-vnz6r" (UID: "b8276ab4-da95-49d7-81ed-bbc401ac406a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:28:15.449658 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.449621 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-113.ec2.internal" event="NodeReady" Aug 04 10:28:15.450234 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.449781 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Aug 04 10:28:15.500013 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.499977 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-597ff7459b-kwzrb"] Aug 04 10:28:15.528531 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.528488 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qjvd4"] Aug 04 10:28:15.528714 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.528622 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:15.532229 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.531850 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Aug 04 10:28:15.532399 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.532299 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wt4lh\"" Aug 04 10:28:15.532602 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.532498 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Aug 04 10:28:15.533462 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.532720 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Aug 04 10:28:15.540023 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.540000 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Aug 04 10:28:15.547610 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.547589 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qbft2"] Aug 04 10:28:15.547735 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.547713 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qjvd4" Aug 04 10:28:15.550524 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.550380 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Aug 04 10:28:15.550524 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.550413 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Aug 04 10:28:15.550524 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.550432 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wlm2z\"" Aug 04 10:28:15.569652 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.569614 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-597ff7459b-kwzrb"] Aug 04 10:28:15.569652 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.569648 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qjvd4"] Aug 04 10:28:15.569863 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.569666 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qbft2"] Aug 04 10:28:15.569863 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.569748 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qbft2" Aug 04 10:28:15.572735 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.572710 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Aug 04 10:28:15.572948 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.572934 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Aug 04 10:28:15.573154 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.573133 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Aug 04 10:28:15.573335 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.573321 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8cx65\"" Aug 04 10:28:15.618578 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.618533 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d4fd3c4-711b-4896-83a7-18dc02e85c7d-tmp-dir\") pod \"dns-default-qjvd4\" (UID: \"2d4fd3c4-711b-4896-83a7-18dc02e85c7d\") " pod="openshift-dns/dns-default-qjvd4" Aug 04 10:28:15.618734 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.618590 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt49s\" (UniqueName: \"kubernetes.io/projected/2d4fd3c4-711b-4896-83a7-18dc02e85c7d-kube-api-access-zt49s\") pod \"dns-default-qjvd4\" (UID: \"2d4fd3c4-711b-4896-83a7-18dc02e85c7d\") " pod="openshift-dns/dns-default-qjvd4" Aug 04 10:28:15.618734 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.618617 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-bound-sa-token\") pod \"image-registry-597ff7459b-kwzrb\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:15.618734 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.618683 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nppgp\" (UniqueName: \"kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-kube-api-access-nppgp\") pod \"image-registry-597ff7459b-kwzrb\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:15.618734 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.618704 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/80278eaf-3e7b-4e5b-a540-8314ecd09e96-registry-certificates\") pod \"image-registry-597ff7459b-kwzrb\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:15.618734 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.618723 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d4fd3c4-711b-4896-83a7-18dc02e85c7d-metrics-tls\") pod \"dns-default-qjvd4\" (UID: \"2d4fd3c4-711b-4896-83a7-18dc02e85c7d\") " pod="openshift-dns/dns-default-qjvd4" Aug 04 10:28:15.618916 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.618783 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-registry-tls\") pod \"image-registry-597ff7459b-kwzrb\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:15.618916 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.618823 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/80278eaf-3e7b-4e5b-a540-8314ecd09e96-ca-trust-extracted\") pod \"image-registry-597ff7459b-kwzrb\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:15.618916 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.618857 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4fd3c4-711b-4896-83a7-18dc02e85c7d-config-volume\") pod \"dns-default-qjvd4\" (UID: \"2d4fd3c4-711b-4896-83a7-18dc02e85c7d\") " pod="openshift-dns/dns-default-qjvd4" Aug 04 10:28:15.618916 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.618882 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/80278eaf-3e7b-4e5b-a540-8314ecd09e96-installation-pull-secrets\") pod \"image-registry-597ff7459b-kwzrb\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:15.618916 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.618904 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80278eaf-3e7b-4e5b-a540-8314ecd09e96-trusted-ca\") pod \"image-registry-597ff7459b-kwzrb\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:15.619073 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.618931 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/80278eaf-3e7b-4e5b-a540-8314ecd09e96-image-registry-private-configuration\") pod \"image-registry-597ff7459b-kwzrb\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:15.719265 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.719227 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80278eaf-3e7b-4e5b-a540-8314ecd09e96-trusted-ca\") pod \"image-registry-597ff7459b-kwzrb\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:15.719444 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.719273 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/80278eaf-3e7b-4e5b-a540-8314ecd09e96-image-registry-private-configuration\") pod \"image-registry-597ff7459b-kwzrb\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:15.719444 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.719369 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ecb558b-6b0a-42a7-a5ae-69fad1241694-cert\") pod \"ingress-canary-qbft2\" (UID: \"6ecb558b-6b0a-42a7-a5ae-69fad1241694\") " pod="openshift-ingress-canary/ingress-canary-qbft2" Aug 04 10:28:15.719444 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.719407 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d4fd3c4-711b-4896-83a7-18dc02e85c7d-tmp-dir\") pod \"dns-default-qjvd4\" (UID: \"2d4fd3c4-711b-4896-83a7-18dc02e85c7d\") " pod="openshift-dns/dns-default-qjvd4" Aug 04 10:28:15.719444 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.719427 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt49s\" (UniqueName: \"kubernetes.io/projected/2d4fd3c4-711b-4896-83a7-18dc02e85c7d-kube-api-access-zt49s\") pod \"dns-default-qjvd4\" (UID: \"2d4fd3c4-711b-4896-83a7-18dc02e85c7d\") " pod="openshift-dns/dns-default-qjvd4" Aug 04 10:28:15.719674 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.719486 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-bound-sa-token\") pod \"image-registry-597ff7459b-kwzrb\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:15.719674 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.719526 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nppgp\" (UniqueName: \"kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-kube-api-access-nppgp\") pod \"image-registry-597ff7459b-kwzrb\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:15.719674 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.719573 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/80278eaf-3e7b-4e5b-a540-8314ecd09e96-registry-certificates\") pod \"image-registry-597ff7459b-kwzrb\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:15.719674 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.719600 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d4fd3c4-711b-4896-83a7-18dc02e85c7d-metrics-tls\") pod \"dns-default-qjvd4\" (UID: \"2d4fd3c4-711b-4896-83a7-18dc02e85c7d\") " pod="openshift-dns/dns-default-qjvd4" Aug 04 10:28:15.719674 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.719626 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssl9r\" (UniqueName: \"kubernetes.io/projected/6ecb558b-6b0a-42a7-a5ae-69fad1241694-kube-api-access-ssl9r\") pod \"ingress-canary-qbft2\" (UID: \"6ecb558b-6b0a-42a7-a5ae-69fad1241694\") " pod="openshift-ingress-canary/ingress-canary-qbft2" Aug 04 10:28:15.719936 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.719682 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-registry-tls\") pod \"image-registry-597ff7459b-kwzrb\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:15.719936 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.719705 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/80278eaf-3e7b-4e5b-a540-8314ecd09e96-ca-trust-extracted\") pod \"image-registry-597ff7459b-kwzrb\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:15.719936 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.719727 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4fd3c4-711b-4896-83a7-18dc02e85c7d-config-volume\") pod \"dns-default-qjvd4\" (UID: \"2d4fd3c4-711b-4896-83a7-18dc02e85c7d\") " pod="openshift-dns/dns-default-qjvd4" Aug 04 10:28:15.719936 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.719763 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/80278eaf-3e7b-4e5b-a540-8314ecd09e96-installation-pull-secrets\") pod \"image-registry-597ff7459b-kwzrb\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:15.719936 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.719804 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d4fd3c4-711b-4896-83a7-18dc02e85c7d-tmp-dir\") pod \"dns-default-qjvd4\" (UID: \"2d4fd3c4-711b-4896-83a7-18dc02e85c7d\") " pod="openshift-dns/dns-default-qjvd4" Aug 04 10:28:15.720176 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:15.719938 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 10:28:15.720176 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:15.720009 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d4fd3c4-711b-4896-83a7-18dc02e85c7d-metrics-tls podName:2d4fd3c4-711b-4896-83a7-18dc02e85c7d nodeName:}" failed. No retries permitted until 2026-08-04 10:28:16.21998724 +0000 UTC m=+35.640874645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2d4fd3c4-711b-4896-83a7-18dc02e85c7d-metrics-tls") pod "dns-default-qjvd4" (UID: "2d4fd3c4-711b-4896-83a7-18dc02e85c7d") : secret "dns-default-metrics-tls" not found Aug 04 10:28:15.720176 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.720127 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/80278eaf-3e7b-4e5b-a540-8314ecd09e96-ca-trust-extracted\") pod \"image-registry-597ff7459b-kwzrb\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:15.720307 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:15.720230 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 10:28:15.720307 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:15.720245 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-597ff7459b-kwzrb: secret "image-registry-tls" not found Aug 04 10:28:15.720307 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.720256 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/80278eaf-3e7b-4e5b-a540-8314ecd09e96-registry-certificates\") pod \"image-registry-597ff7459b-kwzrb\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:15.720307 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:15.720304 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-registry-tls podName:80278eaf-3e7b-4e5b-a540-8314ecd09e96 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:16.220290038 +0000 UTC m=+35.641177441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-registry-tls") pod "image-registry-597ff7459b-kwzrb" (UID: "80278eaf-3e7b-4e5b-a540-8314ecd09e96") : secret "image-registry-tls" not found Aug 04 10:28:15.720474 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.720321 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4fd3c4-711b-4896-83a7-18dc02e85c7d-config-volume\") pod \"dns-default-qjvd4\" (UID: \"2d4fd3c4-711b-4896-83a7-18dc02e85c7d\") " pod="openshift-dns/dns-default-qjvd4" Aug 04 10:28:15.720474 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.720353 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80278eaf-3e7b-4e5b-a540-8314ecd09e96-trusted-ca\") pod \"image-registry-597ff7459b-kwzrb\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:15.723821 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.723800 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/80278eaf-3e7b-4e5b-a540-8314ecd09e96-installation-pull-secrets\") pod \"image-registry-597ff7459b-kwzrb\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:15.723936 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.723806 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/80278eaf-3e7b-4e5b-a540-8314ecd09e96-image-registry-private-configuration\") pod \"image-registry-597ff7459b-kwzrb\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:15.729115 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.728923 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-bound-sa-token\") pod \"image-registry-597ff7459b-kwzrb\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:15.730040 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.730018 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nppgp\" (UniqueName: \"kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-kube-api-access-nppgp\") pod \"image-registry-597ff7459b-kwzrb\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:15.730134 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.730111 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt49s\" (UniqueName: \"kubernetes.io/projected/2d4fd3c4-711b-4896-83a7-18dc02e85c7d-kube-api-access-zt49s\") pod \"dns-default-qjvd4\" (UID: \"2d4fd3c4-711b-4896-83a7-18dc02e85c7d\") " pod="openshift-dns/dns-default-qjvd4" Aug 04 10:28:15.820877 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.820846 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssl9r\" (UniqueName: \"kubernetes.io/projected/6ecb558b-6b0a-42a7-a5ae-69fad1241694-kube-api-access-ssl9r\") pod \"ingress-canary-qbft2\" (UID: \"6ecb558b-6b0a-42a7-a5ae-69fad1241694\") " pod="openshift-ingress-canary/ingress-canary-qbft2" Aug 04 10:28:15.821146 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.820946 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ecb558b-6b0a-42a7-a5ae-69fad1241694-cert\") pod \"ingress-canary-qbft2\" (UID: \"6ecb558b-6b0a-42a7-a5ae-69fad1241694\") " pod="openshift-ingress-canary/ingress-canary-qbft2" Aug 04 10:28:15.824020 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.823997 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ecb558b-6b0a-42a7-a5ae-69fad1241694-cert\") pod \"ingress-canary-qbft2\" (UID: \"6ecb558b-6b0a-42a7-a5ae-69fad1241694\") " pod="openshift-ingress-canary/ingress-canary-qbft2" Aug 04 10:28:15.831390 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.831366 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssl9r\" (UniqueName: \"kubernetes.io/projected/6ecb558b-6b0a-42a7-a5ae-69fad1241694-kube-api-access-ssl9r\") pod \"ingress-canary-qbft2\" (UID: \"6ecb558b-6b0a-42a7-a5ae-69fad1241694\") " pod="openshift-ingress-canary/ingress-canary-qbft2" Aug 04 10:28:15.881454 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:15.881411 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qbft2" Aug 04 10:28:16.018777 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:16.018743 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qbft2"] Aug 04 10:28:16.023476 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:28:16.023447 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ecb558b_6b0a_42a7_a5ae_69fad1241694.slice/crio-d2b9f0865562a1988493dc875d4e9c2bcf94b82feee4aaacc516f628cad56a1b WatchSource:0}: Error finding container d2b9f0865562a1988493dc875d4e9c2bcf94b82feee4aaacc516f628cad56a1b: Status 404 returned error can't find the container with id d2b9f0865562a1988493dc875d4e9c2bcf94b82feee4aaacc516f628cad56a1b Aug 04 10:28:16.094270 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:16.094238 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:28:16.094270 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:16.094260 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnz6r" Aug 04 10:28:16.094520 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:16.094240 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc2fl" Aug 04 10:28:16.097835 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:16.097814 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-p48mn\"" Aug 04 10:28:16.097835 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:16.097824 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-clvkl\"" Aug 04 10:28:16.098030 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:16.097814 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 04 10:28:16.098030 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:16.097849 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 04 10:28:16.098030 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:16.097814 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 10:28:16.098030 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:16.097833 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 10:28:16.224386 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:16.224296 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-registry-tls\") pod \"image-registry-597ff7459b-kwzrb\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:16.224386 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:16.224371 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d4fd3c4-711b-4896-83a7-18dc02e85c7d-metrics-tls\") pod \"dns-default-qjvd4\" (UID: \"2d4fd3c4-711b-4896-83a7-18dc02e85c7d\") " pod="openshift-dns/dns-default-qjvd4" Aug 04 10:28:16.224645 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:16.224453 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 10:28:16.224645 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:16.224459 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 10:28:16.224645 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:16.224479 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-597ff7459b-kwzrb: secret "image-registry-tls" not found Aug 04 10:28:16.224645 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:16.224505 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d4fd3c4-711b-4896-83a7-18dc02e85c7d-metrics-tls podName:2d4fd3c4-711b-4896-83a7-18dc02e85c7d nodeName:}" failed. No retries permitted until 2026-08-04 10:28:17.224492973 +0000 UTC m=+36.645380363 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2d4fd3c4-711b-4896-83a7-18dc02e85c7d-metrics-tls") pod "dns-default-qjvd4" (UID: "2d4fd3c4-711b-4896-83a7-18dc02e85c7d") : secret "dns-default-metrics-tls" not found Aug 04 10:28:16.224645 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:16.224530 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-registry-tls podName:80278eaf-3e7b-4e5b-a540-8314ecd09e96 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:17.224512775 +0000 UTC m=+36.645400165 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-registry-tls") pod "image-registry-597ff7459b-kwzrb" (UID: "80278eaf-3e7b-4e5b-a540-8314ecd09e96") : secret "image-registry-tls" not found Aug 04 10:28:16.284081 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:16.284044 2580 generic.go:356] "Generic (PLEG): container finished" podID="13819f5d-21f5-43f5-88f6-8bd915c7e86f" containerID="3758795163cdeb058c2517cbd833a496f9673eccdd3ee3f61ef757bfb787e4ab" exitCode=0 Aug 04 10:28:16.284231 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:16.284111 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r66c8" event={"ID":"13819f5d-21f5-43f5-88f6-8bd915c7e86f","Type":"ContainerDied","Data":"3758795163cdeb058c2517cbd833a496f9673eccdd3ee3f61ef757bfb787e4ab"} Aug 04 10:28:16.287083 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:16.286311 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qbft2" event={"ID":"6ecb558b-6b0a-42a7-a5ae-69fad1241694","Type":"ContainerStarted","Data":"d2b9f0865562a1988493dc875d4e9c2bcf94b82feee4aaacc516f628cad56a1b"} Aug 04 10:28:17.231745 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:17.231704 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d4fd3c4-711b-4896-83a7-18dc02e85c7d-metrics-tls\") pod \"dns-default-qjvd4\" (UID: \"2d4fd3c4-711b-4896-83a7-18dc02e85c7d\") " pod="openshift-dns/dns-default-qjvd4" Aug 04 10:28:17.232249 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:17.231782 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-registry-tls\") pod \"image-registry-597ff7459b-kwzrb\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:17.232249 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:17.231875 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 10:28:17.232249 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:17.231907 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 10:28:17.232249 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:17.231922 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-597ff7459b-kwzrb: secret "image-registry-tls" not found Aug 04 10:28:17.232249 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:17.231954 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d4fd3c4-711b-4896-83a7-18dc02e85c7d-metrics-tls podName:2d4fd3c4-711b-4896-83a7-18dc02e85c7d nodeName:}" failed. No retries permitted until 2026-08-04 10:28:19.231935221 +0000 UTC m=+38.652822610 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2d4fd3c4-711b-4896-83a7-18dc02e85c7d-metrics-tls") pod "dns-default-qjvd4" (UID: "2d4fd3c4-711b-4896-83a7-18dc02e85c7d") : secret "dns-default-metrics-tls" not found Aug 04 10:28:17.232249 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:17.231972 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-registry-tls podName:80278eaf-3e7b-4e5b-a540-8314ecd09e96 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:19.231964091 +0000 UTC m=+38.652851487 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-registry-tls") pod "image-registry-597ff7459b-kwzrb" (UID: "80278eaf-3e7b-4e5b-a540-8314ecd09e96") : secret "image-registry-tls" not found Aug 04 10:28:17.292344 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:17.292309 2580 generic.go:356] "Generic (PLEG): container finished" podID="13819f5d-21f5-43f5-88f6-8bd915c7e86f" containerID="8c8e1671653ae3ed4fe2a0f887be7511e78c110646ab05787793ada771995057" exitCode=0 Aug 04 10:28:17.292515 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:17.292357 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r66c8" event={"ID":"13819f5d-21f5-43f5-88f6-8bd915c7e86f","Type":"ContainerDied","Data":"8c8e1671653ae3ed4fe2a0f887be7511e78c110646ab05787793ada771995057"} Aug 04 10:28:18.298701 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:18.298500 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r66c8" event={"ID":"13819f5d-21f5-43f5-88f6-8bd915c7e86f","Type":"ContainerStarted","Data":"589109880e5db08a3e6d13180b7209e556cc7397eb58498bb76d47b0901c4107"} Aug 04 10:28:18.323756 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:18.323701 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-r66c8" podStartSLOduration=7.964709638 podStartE2EDuration="37.323678117s" totalImagesPullingTime="29.358968479s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:27:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:18.321790568 +0000 UTC m=+37.742677980" watchObservedRunningTime="2026-08-04 10:28:18.323678117 +0000 UTC m=+37.744565529" Aug 04 10:28:19.246743 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:19.246709 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d4fd3c4-711b-4896-83a7-18dc02e85c7d-metrics-tls\") pod \"dns-default-qjvd4\" (UID: \"2d4fd3c4-711b-4896-83a7-18dc02e85c7d\") " pod="openshift-dns/dns-default-qjvd4" Aug 04 10:28:19.246903 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:19.246761 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-registry-tls\") pod \"image-registry-597ff7459b-kwzrb\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:19.246903 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:19.246868 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 10:28:19.246973 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:19.246932 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d4fd3c4-711b-4896-83a7-18dc02e85c7d-metrics-tls podName:2d4fd3c4-711b-4896-83a7-18dc02e85c7d nodeName:}" failed. No retries permitted until 2026-08-04 10:28:23.246916927 +0000 UTC m=+42.667804316 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2d4fd3c4-711b-4896-83a7-18dc02e85c7d-metrics-tls") pod "dns-default-qjvd4" (UID: "2d4fd3c4-711b-4896-83a7-18dc02e85c7d") : secret "dns-default-metrics-tls" not found Aug 04 10:28:19.246973 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:19.246872 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 10:28:19.246973 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:19.246959 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-597ff7459b-kwzrb: secret "image-registry-tls" not found Aug 04 10:28:19.247071 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:19.246999 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-registry-tls podName:80278eaf-3e7b-4e5b-a540-8314ecd09e96 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:23.246988141 +0000 UTC m=+42.667875530 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-registry-tls") pod "image-registry-597ff7459b-kwzrb" (UID: "80278eaf-3e7b-4e5b-a540-8314ecd09e96") : secret "image-registry-tls" not found Aug 04 10:28:19.302385 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:19.302353 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qbft2" event={"ID":"6ecb558b-6b0a-42a7-a5ae-69fad1241694","Type":"ContainerStarted","Data":"1887bc6f10fd0fb6869137db050e7d29b46fa7767f8a63c7f13f4acafd72498c"} Aug 04 10:28:19.323189 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:19.323136 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qbft2" podStartSLOduration=1.901015116 podStartE2EDuration="4.323121479s" totalImagesPullingTime="2.422106363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:28:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:19.321345875 +0000 UTC m=+38.742233285" watchObservedRunningTime="2026-08-04 10:28:19.323121479 +0000 UTC m=+38.744008890" Aug 04 10:28:22.069629 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:22.069589 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-k4lhp"] Aug 04 10:28:22.097008 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:22.096978 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k4lhp"] Aug 04 10:28:22.097178 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:22.097108 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k4lhp" Aug 04 10:28:22.099939 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:22.099916 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Aug 04 10:28:22.100058 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:22.099986 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Aug 04 10:28:22.100309 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:22.100291 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Aug 04 10:28:22.100390 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:22.100321 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Aug 04 10:28:22.100390 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:22.100332 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ntlkp\"" Aug 04 10:28:22.271735 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:22.271697 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k4lhp\" (UID: \"5c6f919e-0e11-4f1d-a2f0-1db97b80e53d\") " pod="openshift-insights/insights-runtime-extractor-k4lhp" Aug 04 10:28:22.271735 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:22.271735 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k4lhp\" (UID: \"5c6f919e-0e11-4f1d-a2f0-1db97b80e53d\") " pod="openshift-insights/insights-runtime-extractor-k4lhp" Aug 04 10:28:22.271941 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:22.271774 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-crio-socket\") pod \"insights-runtime-extractor-k4lhp\" (UID: \"5c6f919e-0e11-4f1d-a2f0-1db97b80e53d\") " pod="openshift-insights/insights-runtime-extractor-k4lhp" Aug 04 10:28:22.271941 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:22.271824 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-data-volume\") pod \"insights-runtime-extractor-k4lhp\" (UID: \"5c6f919e-0e11-4f1d-a2f0-1db97b80e53d\") " pod="openshift-insights/insights-runtime-extractor-k4lhp" Aug 04 10:28:22.271941 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:22.271839 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rdln\" (UniqueName: \"kubernetes.io/projected/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-kube-api-access-7rdln\") pod \"insights-runtime-extractor-k4lhp\" (UID: \"5c6f919e-0e11-4f1d-a2f0-1db97b80e53d\") " pod="openshift-insights/insights-runtime-extractor-k4lhp" Aug 04 10:28:22.372596 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:22.372547 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-crio-socket\") pod \"insights-runtime-extractor-k4lhp\" (UID: \"5c6f919e-0e11-4f1d-a2f0-1db97b80e53d\") " pod="openshift-insights/insights-runtime-extractor-k4lhp" Aug 04 10:28:22.372741 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:22.372611 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-data-volume\") pod \"insights-runtime-extractor-k4lhp\" (UID: \"5c6f919e-0e11-4f1d-a2f0-1db97b80e53d\") " pod="openshift-insights/insights-runtime-extractor-k4lhp" Aug 04 10:28:22.372741 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:22.372638 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rdln\" (UniqueName: \"kubernetes.io/projected/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-kube-api-access-7rdln\") pod \"insights-runtime-extractor-k4lhp\" (UID: \"5c6f919e-0e11-4f1d-a2f0-1db97b80e53d\") " pod="openshift-insights/insights-runtime-extractor-k4lhp" Aug 04 10:28:22.372741 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:22.372715 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k4lhp\" (UID: \"5c6f919e-0e11-4f1d-a2f0-1db97b80e53d\") " pod="openshift-insights/insights-runtime-extractor-k4lhp" Aug 04 10:28:22.372741 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:22.372737 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k4lhp\" (UID: \"5c6f919e-0e11-4f1d-a2f0-1db97b80e53d\") " pod="openshift-insights/insights-runtime-extractor-k4lhp" Aug 04 10:28:22.372929 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:22.372754 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-crio-socket\") pod \"insights-runtime-extractor-k4lhp\" (UID: \"5c6f919e-0e11-4f1d-a2f0-1db97b80e53d\") " pod="openshift-insights/insights-runtime-extractor-k4lhp" Aug 04 10:28:22.372929 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:22.372826 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 10:28:22.372929 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:22.372845 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 10:28:22.372929 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:22.372890 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-kube-rbac-proxy-cm podName:5c6f919e-0e11-4f1d-a2f0-1db97b80e53d nodeName:}" failed. No retries permitted until 2026-08-04 10:28:22.872874256 +0000 UTC m=+42.293761645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k4lhp" (UID: "5c6f919e-0e11-4f1d-a2f0-1db97b80e53d") : configmap "kube-rbac-proxy" not found Aug 04 10:28:22.372929 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:22.372905 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-insights-runtime-extractor-tls podName:5c6f919e-0e11-4f1d-a2f0-1db97b80e53d nodeName:}" failed. No retries permitted until 2026-08-04 10:28:22.872899679 +0000 UTC m=+42.293787068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k4lhp" (UID: "5c6f919e-0e11-4f1d-a2f0-1db97b80e53d") : secret "insights-runtime-extractor-tls" not found Aug 04 10:28:22.373161 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:22.372986 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-data-volume\") pod \"insights-runtime-extractor-k4lhp\" (UID: \"5c6f919e-0e11-4f1d-a2f0-1db97b80e53d\") " pod="openshift-insights/insights-runtime-extractor-k4lhp" Aug 04 10:28:22.386343 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:22.386314 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rdln\" (UniqueName: \"kubernetes.io/projected/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-kube-api-access-7rdln\") pod \"insights-runtime-extractor-k4lhp\" (UID: \"5c6f919e-0e11-4f1d-a2f0-1db97b80e53d\") " pod="openshift-insights/insights-runtime-extractor-k4lhp" Aug 04 10:28:22.877403 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:22.877369 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k4lhp\" (UID: \"5c6f919e-0e11-4f1d-a2f0-1db97b80e53d\") " pod="openshift-insights/insights-runtime-extractor-k4lhp" Aug 04 10:28:22.877554 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:22.877410 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k4lhp\" (UID: \"5c6f919e-0e11-4f1d-a2f0-1db97b80e53d\") " pod="openshift-insights/insights-runtime-extractor-k4lhp" Aug 04 10:28:22.877554 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:22.877497 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 10:28:22.877661 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:22.877583 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 10:28:22.877661 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:22.877599 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-kube-rbac-proxy-cm podName:5c6f919e-0e11-4f1d-a2f0-1db97b80e53d nodeName:}" failed. No retries permitted until 2026-08-04 10:28:23.877553168 +0000 UTC m=+43.298440557 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k4lhp" (UID: "5c6f919e-0e11-4f1d-a2f0-1db97b80e53d") : configmap "kube-rbac-proxy" not found Aug 04 10:28:22.877661 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:22.877621 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-insights-runtime-extractor-tls podName:5c6f919e-0e11-4f1d-a2f0-1db97b80e53d nodeName:}" failed. No retries permitted until 2026-08-04 10:28:23.877609225 +0000 UTC m=+43.298496634 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k4lhp" (UID: "5c6f919e-0e11-4f1d-a2f0-1db97b80e53d") : secret "insights-runtime-extractor-tls" not found Aug 04 10:28:23.281486 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:23.281403 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d4fd3c4-711b-4896-83a7-18dc02e85c7d-metrics-tls\") pod \"dns-default-qjvd4\" (UID: \"2d4fd3c4-711b-4896-83a7-18dc02e85c7d\") " pod="openshift-dns/dns-default-qjvd4" Aug 04 10:28:23.281486 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:23.281459 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-registry-tls\") pod \"image-registry-597ff7459b-kwzrb\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:23.281871 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:23.281578 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 10:28:23.281871 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:23.281609 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 10:28:23.281871 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:23.281621 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-597ff7459b-kwzrb: secret "image-registry-tls" not found Aug 04 10:28:23.281871 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:23.281640 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d4fd3c4-711b-4896-83a7-18dc02e85c7d-metrics-tls podName:2d4fd3c4-711b-4896-83a7-18dc02e85c7d nodeName:}" failed. No retries permitted until 2026-08-04 10:28:31.281624551 +0000 UTC m=+50.702511940 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2d4fd3c4-711b-4896-83a7-18dc02e85c7d-metrics-tls") pod "dns-default-qjvd4" (UID: "2d4fd3c4-711b-4896-83a7-18dc02e85c7d") : secret "dns-default-metrics-tls" not found Aug 04 10:28:23.281871 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:23.281672 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-registry-tls podName:80278eaf-3e7b-4e5b-a540-8314ecd09e96 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:31.28166011 +0000 UTC m=+50.702547499 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-registry-tls") pod "image-registry-597ff7459b-kwzrb" (UID: "80278eaf-3e7b-4e5b-a540-8314ecd09e96") : secret "image-registry-tls" not found Aug 04 10:28:23.802971 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:23.802942 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m4sfm_59e9a581-bf1d-4594-8d5b-9dba10576132/dns-node-resolver/0.log" Aug 04 10:28:23.886408 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:23.886363 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k4lhp\" (UID: \"5c6f919e-0e11-4f1d-a2f0-1db97b80e53d\") " pod="openshift-insights/insights-runtime-extractor-k4lhp" Aug 04 10:28:23.886408 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:23.886408 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k4lhp\" (UID: \"5c6f919e-0e11-4f1d-a2f0-1db97b80e53d\") " pod="openshift-insights/insights-runtime-extractor-k4lhp" Aug 04 10:28:23.886620 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:23.886498 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 10:28:23.886620 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:23.886580 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-kube-rbac-proxy-cm podName:5c6f919e-0e11-4f1d-a2f0-1db97b80e53d nodeName:}" failed. No retries permitted until 2026-08-04 10:28:25.886544739 +0000 UTC m=+45.307432147 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k4lhp" (UID: "5c6f919e-0e11-4f1d-a2f0-1db97b80e53d") : configmap "kube-rbac-proxy" not found Aug 04 10:28:23.886692 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:23.886636 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 10:28:23.886724 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:23.886700 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-insights-runtime-extractor-tls podName:5c6f919e-0e11-4f1d-a2f0-1db97b80e53d nodeName:}" failed. No retries permitted until 2026-08-04 10:28:25.886687335 +0000 UTC m=+45.307574724 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k4lhp" (UID: "5c6f919e-0e11-4f1d-a2f0-1db97b80e53d") : secret "insights-runtime-extractor-tls" not found Aug 04 10:28:24.803361 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:24.803335 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bhq6d_53b096bc-7df7-4067-8c04-fd4d68746d12/node-ca/0.log" Aug 04 10:28:25.605048 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:25.604987 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qbft2_6ecb558b-6b0a-42a7-a5ae-69fad1241694/serve-healthcheck-canary/0.log" Aug 04 10:28:25.902332 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:25.902242 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k4lhp\" (UID: \"5c6f919e-0e11-4f1d-a2f0-1db97b80e53d\") " pod="openshift-insights/insights-runtime-extractor-k4lhp" Aug 04 10:28:25.902332 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:25.902278 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k4lhp\" (UID: \"5c6f919e-0e11-4f1d-a2f0-1db97b80e53d\") " pod="openshift-insights/insights-runtime-extractor-k4lhp" Aug 04 10:28:25.902777 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:25.902372 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 10:28:25.902777 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:25.902384 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 10:28:25.902777 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:25.902435 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-insights-runtime-extractor-tls podName:5c6f919e-0e11-4f1d-a2f0-1db97b80e53d nodeName:}" failed. No retries permitted until 2026-08-04 10:28:29.902422079 +0000 UTC m=+49.323309467 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k4lhp" (UID: "5c6f919e-0e11-4f1d-a2f0-1db97b80e53d") : secret "insights-runtime-extractor-tls" not found Aug 04 10:28:25.902777 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:25.902449 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-kube-rbac-proxy-cm podName:5c6f919e-0e11-4f1d-a2f0-1db97b80e53d nodeName:}" failed. No retries permitted until 2026-08-04 10:28:29.902442857 +0000 UTC m=+49.323330259 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k4lhp" (UID: "5c6f919e-0e11-4f1d-a2f0-1db97b80e53d") : configmap "kube-rbac-proxy" not found Aug 04 10:28:29.933540 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:29.933496 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k4lhp\" (UID: \"5c6f919e-0e11-4f1d-a2f0-1db97b80e53d\") " pod="openshift-insights/insights-runtime-extractor-k4lhp" Aug 04 10:28:29.933540 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:29.933538 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k4lhp\" (UID: \"5c6f919e-0e11-4f1d-a2f0-1db97b80e53d\") " pod="openshift-insights/insights-runtime-extractor-k4lhp" Aug 04 10:28:29.934014 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:29.933651 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 10:28:29.934014 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:29.933674 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 10:28:29.934014 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:29.933736 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-insights-runtime-extractor-tls podName:5c6f919e-0e11-4f1d-a2f0-1db97b80e53d nodeName:}" failed. No retries permitted until 2026-08-04 10:28:37.933718144 +0000 UTC m=+57.354605537 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k4lhp" (UID: "5c6f919e-0e11-4f1d-a2f0-1db97b80e53d") : secret "insights-runtime-extractor-tls" not found Aug 04 10:28:29.934014 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:29.933754 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-kube-rbac-proxy-cm podName:5c6f919e-0e11-4f1d-a2f0-1db97b80e53d nodeName:}" failed. No retries permitted until 2026-08-04 10:28:37.933744964 +0000 UTC m=+57.354632353 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k4lhp" (UID: "5c6f919e-0e11-4f1d-a2f0-1db97b80e53d") : configmap "kube-rbac-proxy" not found Aug 04 10:28:31.344912 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:31.344875 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-registry-tls\") pod \"image-registry-597ff7459b-kwzrb\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:31.345328 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:31.344980 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d4fd3c4-711b-4896-83a7-18dc02e85c7d-metrics-tls\") pod \"dns-default-qjvd4\" (UID: \"2d4fd3c4-711b-4896-83a7-18dc02e85c7d\") " pod="openshift-dns/dns-default-qjvd4" Aug 04 10:28:31.345328 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:31.345032 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 10:28:31.345328 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:31.345050 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-597ff7459b-kwzrb: secret "image-registry-tls" not found Aug 04 10:28:31.345328 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:31.345075 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 10:28:31.345328 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:31.345104 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-registry-tls podName:80278eaf-3e7b-4e5b-a540-8314ecd09e96 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:47.345087859 +0000 UTC m=+66.765975269 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-registry-tls") pod "image-registry-597ff7459b-kwzrb" (UID: "80278eaf-3e7b-4e5b-a540-8314ecd09e96") : secret "image-registry-tls" not found Aug 04 10:28:31.345328 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:31.345122 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d4fd3c4-711b-4896-83a7-18dc02e85c7d-metrics-tls podName:2d4fd3c4-711b-4896-83a7-18dc02e85c7d nodeName:}" failed. No retries permitted until 2026-08-04 10:28:47.345109136 +0000 UTC m=+66.765996525 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2d4fd3c4-711b-4896-83a7-18dc02e85c7d-metrics-tls") pod "dns-default-qjvd4" (UID: "2d4fd3c4-711b-4896-83a7-18dc02e85c7d") : secret "dns-default-metrics-tls" not found Aug 04 10:28:37.998099 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:37.998038 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k4lhp\" (UID: \"5c6f919e-0e11-4f1d-a2f0-1db97b80e53d\") " pod="openshift-insights/insights-runtime-extractor-k4lhp" Aug 04 10:28:37.998099 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:37.998098 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k4lhp\" (UID: \"5c6f919e-0e11-4f1d-a2f0-1db97b80e53d\") " pod="openshift-insights/insights-runtime-extractor-k4lhp" Aug 04 10:28:37.998682 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:37.998210 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 10:28:37.998682 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:28:37.998271 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-insights-runtime-extractor-tls podName:5c6f919e-0e11-4f1d-a2f0-1db97b80e53d nodeName:}" failed. No retries permitted until 2026-08-04 10:28:53.998257833 +0000 UTC m=+73.419145222 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k4lhp" (UID: "5c6f919e-0e11-4f1d-a2f0-1db97b80e53d") : secret "insights-runtime-extractor-tls" not found Aug 04 10:28:37.998682 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:37.998654 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k4lhp\" (UID: \"5c6f919e-0e11-4f1d-a2f0-1db97b80e53d\") " pod="openshift-insights/insights-runtime-extractor-k4lhp" Aug 04 10:28:42.282315 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:42.282287 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tb4zk" Aug 04 10:28:46.861522 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:46.861409 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0db48901-2221-480e-97b0-cd6be997aaef-metrics-certs\") pod \"network-metrics-daemon-wc2fl\" (UID: \"0db48901-2221-480e-97b0-cd6be997aaef\") " pod="openshift-multus/network-metrics-daemon-wc2fl" Aug 04 10:28:46.861917 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:46.861584 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4tpg\" (UniqueName: \"kubernetes.io/projected/b8276ab4-da95-49d7-81ed-bbc401ac406a-kube-api-access-g4tpg\") pod \"network-check-target-vnz6r\" (UID: \"b8276ab4-da95-49d7-81ed-bbc401ac406a\") " pod="openshift-network-diagnostics/network-check-target-vnz6r" Aug 04 10:28:46.861917 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:46.861604 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0a98e70-0e23-4348-9b6d-93b4081069df-original-pull-secret\") pod \"global-pull-secret-syncer-rzhxt\" (UID: \"b0a98e70-0e23-4348-9b6d-93b4081069df\") " pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:28:46.864512 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:46.864484 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 04 10:28:46.864512 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:46.864509 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 10:28:46.865781 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:46.865763 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 10:28:46.874458 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:46.874418 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0a98e70-0e23-4348-9b6d-93b4081069df-original-pull-secret\") pod \"global-pull-secret-syncer-rzhxt\" (UID: \"b0a98e70-0e23-4348-9b6d-93b4081069df\") " pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:28:46.874458 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:46.874439 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0db48901-2221-480e-97b0-cd6be997aaef-metrics-certs\") pod \"network-metrics-daemon-wc2fl\" (UID: \"0db48901-2221-480e-97b0-cd6be997aaef\") " pod="openshift-multus/network-metrics-daemon-wc2fl" Aug 04 10:28:46.874623 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:46.874594 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 04 10:28:46.884497 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:46.884466 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4tpg\" (UniqueName: \"kubernetes.io/projected/b8276ab4-da95-49d7-81ed-bbc401ac406a-kube-api-access-g4tpg\") pod \"network-check-target-vnz6r\" (UID: \"b8276ab4-da95-49d7-81ed-bbc401ac406a\") " pod="openshift-network-diagnostics/network-check-target-vnz6r" Aug 04 10:28:47.005580 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:47.005520 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhxt" Aug 04 10:28:47.013318 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:47.013298 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-clvkl\"" Aug 04 10:28:47.017734 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:47.017705 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-p48mn\"" Aug 04 10:28:47.020211 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:47.020181 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnz6r" Aug 04 10:28:47.024453 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:47.024432 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc2fl" Aug 04 10:28:47.190550 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:47.190521 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vnz6r"] Aug 04 10:28:47.194044 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:47.193203 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rzhxt"] Aug 04 10:28:47.195928 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:28:47.195883 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8276ab4_da95_49d7_81ed_bbc401ac406a.slice/crio-983e8d04c95f48c423e0e5bb8859c2102b542420fa451bad6ba4d98599542feb WatchSource:0}: Error finding container 983e8d04c95f48c423e0e5bb8859c2102b542420fa451bad6ba4d98599542feb: Status 404 returned error can't find the container with id 983e8d04c95f48c423e0e5bb8859c2102b542420fa451bad6ba4d98599542feb Aug 04 10:28:47.198066 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:28:47.198045 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a98e70_0e23_4348_9b6d_93b4081069df.slice/crio-4cf73e98c4511cd62c2573138b18484ea471ec448ec982ccb7dbb5482377540b WatchSource:0}: Error finding container 4cf73e98c4511cd62c2573138b18484ea471ec448ec982ccb7dbb5482377540b: Status 404 returned error can't find the container with id 4cf73e98c4511cd62c2573138b18484ea471ec448ec982ccb7dbb5482377540b Aug 04 10:28:47.210710 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:47.210680 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wc2fl"] Aug 04 10:28:47.213725 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:28:47.213696 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db48901_2221_480e_97b0_cd6be997aaef.slice/crio-9bf4b51baeec52d1faf5a4377ee17f822f7f2af6a1d0eab271466ee2f07e85d5 WatchSource:0}: Error finding container 9bf4b51baeec52d1faf5a4377ee17f822f7f2af6a1d0eab271466ee2f07e85d5: Status 404 returned error can't find the container with id 9bf4b51baeec52d1faf5a4377ee17f822f7f2af6a1d0eab271466ee2f07e85d5 Aug 04 10:28:47.366221 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:47.366139 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d4fd3c4-711b-4896-83a7-18dc02e85c7d-metrics-tls\") pod \"dns-default-qjvd4\" (UID: \"2d4fd3c4-711b-4896-83a7-18dc02e85c7d\") " pod="openshift-dns/dns-default-qjvd4" Aug 04 10:28:47.366221 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:47.366180 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-registry-tls\") pod \"image-registry-597ff7459b-kwzrb\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:47.368693 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:47.368664 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d4fd3c4-711b-4896-83a7-18dc02e85c7d-metrics-tls\") pod \"dns-default-qjvd4\" (UID: \"2d4fd3c4-711b-4896-83a7-18dc02e85c7d\") " pod="openshift-dns/dns-default-qjvd4" Aug 04 10:28:47.368821 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:47.368739 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-registry-tls\") pod \"image-registry-597ff7459b-kwzrb\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:47.371309 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:47.371275 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vnz6r" event={"ID":"b8276ab4-da95-49d7-81ed-bbc401ac406a","Type":"ContainerStarted","Data":"983e8d04c95f48c423e0e5bb8859c2102b542420fa451bad6ba4d98599542feb"} Aug 04 10:28:47.372352 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:47.372332 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wc2fl" event={"ID":"0db48901-2221-480e-97b0-cd6be997aaef","Type":"ContainerStarted","Data":"9bf4b51baeec52d1faf5a4377ee17f822f7f2af6a1d0eab271466ee2f07e85d5"} Aug 04 10:28:47.373369 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:47.373352 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rzhxt" event={"ID":"b0a98e70-0e23-4348-9b6d-93b4081069df","Type":"ContainerStarted","Data":"4cf73e98c4511cd62c2573138b18484ea471ec448ec982ccb7dbb5482377540b"} Aug 04 10:28:47.645518 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:47.645427 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wt4lh\"" Aug 04 10:28:47.652088 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:47.652067 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:47.661841 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:47.661814 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wlm2z\"" Aug 04 10:28:47.668615 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:47.668589 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qjvd4" Aug 04 10:28:47.821532 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:47.821501 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-597ff7459b-kwzrb"] Aug 04 10:28:47.824873 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:28:47.824829 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80278eaf_3e7b_4e5b_a540_8314ecd09e96.slice/crio-fdfec6049eee8f6faca7bb365a9988bf664edcfa9ec962c22b32d92dd9670c91 WatchSource:0}: Error finding container fdfec6049eee8f6faca7bb365a9988bf664edcfa9ec962c22b32d92dd9670c91: Status 404 returned error can't find the container with id fdfec6049eee8f6faca7bb365a9988bf664edcfa9ec962c22b32d92dd9670c91 Aug 04 10:28:47.838812 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:47.838764 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qjvd4"] Aug 04 10:28:47.842164 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:28:47.842136 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d4fd3c4_711b_4896_83a7_18dc02e85c7d.slice/crio-e5cb31aaa89f244fa93a8f5d58be7aaa245732ab68264e9e93a5a312b01926ba WatchSource:0}: Error finding container e5cb31aaa89f244fa93a8f5d58be7aaa245732ab68264e9e93a5a312b01926ba: Status 404 returned error can't find the container with id e5cb31aaa89f244fa93a8f5d58be7aaa245732ab68264e9e93a5a312b01926ba Aug 04 10:28:48.379188 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.379145 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" event={"ID":"80278eaf-3e7b-4e5b-a540-8314ecd09e96","Type":"ContainerStarted","Data":"076fb34a4b23c6e77e0d4c1aafec6fd95b996594c61088901b35a55875aebf1d"} Aug 04 10:28:48.379692 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.379237 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" event={"ID":"80278eaf-3e7b-4e5b-a540-8314ecd09e96","Type":"ContainerStarted","Data":"fdfec6049eee8f6faca7bb365a9988bf664edcfa9ec962c22b32d92dd9670c91"} Aug 04 10:28:48.379951 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.379921 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:28:48.381466 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.381390 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qjvd4" event={"ID":"2d4fd3c4-711b-4896-83a7-18dc02e85c7d","Type":"ContainerStarted","Data":"e5cb31aaa89f244fa93a8f5d58be7aaa245732ab68264e9e93a5a312b01926ba"} Aug 04 10:28:48.646127 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.646017 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" podStartSLOduration=33.645992721 podStartE2EDuration="33.645992721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:48.407305206 +0000 UTC m=+67.828192622" watchObservedRunningTime="2026-08-04 10:28:48.645992721 +0000 UTC m=+68.066880133" Aug 04 10:28:48.647224 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.647193 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-665fbd9b58-4hzt6"] Aug 04 10:28:48.656713 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.656679 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c6d75b56-hfrpb"] Aug 04 10:28:48.656889 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.656833 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-665fbd9b58-4hzt6" Aug 04 10:28:48.659648 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.659616 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Aug 04 10:28:48.659783 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.659658 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-hxf49\"" Aug 04 10:28:48.662141 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.661935 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Aug 04 10:28:48.662141 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.661998 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Aug 04 10:28:48.662141 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.662138 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Aug 04 10:28:48.670155 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.669409 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-665fbd9b58-4hzt6"] Aug 04 10:28:48.670155 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.669447 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-597ff7459b-kwzrb"] Aug 04 10:28:48.670155 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.669588 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c6d75b56-hfrpb" Aug 04 10:28:48.670155 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.670107 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c6d75b56-hfrpb"] Aug 04 10:28:48.673571 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.673412 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Aug 04 10:28:48.673735 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.673696 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Aug 04 10:28:48.673873 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.673851 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Aug 04 10:28:48.674028 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.674005 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Aug 04 10:28:48.680878 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.680851 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/165c32fb-4eac-42f0-88de-d6fe0453ad99-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-665fbd9b58-4hzt6\" (UID: \"165c32fb-4eac-42f0-88de-d6fe0453ad99\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-665fbd9b58-4hzt6" Aug 04 10:28:48.681006 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.680968 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8h9k\" (UniqueName: \"kubernetes.io/projected/165c32fb-4eac-42f0-88de-d6fe0453ad99-kube-api-access-g8h9k\") pod \"managed-serviceaccount-addon-agent-665fbd9b58-4hzt6\" (UID: \"165c32fb-4eac-42f0-88de-d6fe0453ad99\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-665fbd9b58-4hzt6" Aug 04 10:28:48.782014 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.781978 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd41804f-6c04-4553-92c5-1d6b48572f5b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-56c6d75b56-hfrpb\" (UID: \"cd41804f-6c04-4553-92c5-1d6b48572f5b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c6d75b56-hfrpb" Aug 04 10:28:48.782214 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.782023 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6wmk\" (UniqueName: \"kubernetes.io/projected/cd41804f-6c04-4553-92c5-1d6b48572f5b-kube-api-access-j6wmk\") pod \"cluster-proxy-proxy-agent-56c6d75b56-hfrpb\" (UID: \"cd41804f-6c04-4553-92c5-1d6b48572f5b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c6d75b56-hfrpb" Aug 04 10:28:48.782214 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.782066 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/165c32fb-4eac-42f0-88de-d6fe0453ad99-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-665fbd9b58-4hzt6\" (UID: \"165c32fb-4eac-42f0-88de-d6fe0453ad99\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-665fbd9b58-4hzt6" Aug 04 10:28:48.782214 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.782099 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8h9k\" (UniqueName: \"kubernetes.io/projected/165c32fb-4eac-42f0-88de-d6fe0453ad99-kube-api-access-g8h9k\") pod \"managed-serviceaccount-addon-agent-665fbd9b58-4hzt6\" (UID: \"165c32fb-4eac-42f0-88de-d6fe0453ad99\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-665fbd9b58-4hzt6" Aug 04 10:28:48.782214 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.782145 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd41804f-6c04-4553-92c5-1d6b48572f5b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-56c6d75b56-hfrpb\" (UID: \"cd41804f-6c04-4553-92c5-1d6b48572f5b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c6d75b56-hfrpb" Aug 04 10:28:48.782214 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.782192 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cd41804f-6c04-4553-92c5-1d6b48572f5b-ca\") pod \"cluster-proxy-proxy-agent-56c6d75b56-hfrpb\" (UID: \"cd41804f-6c04-4553-92c5-1d6b48572f5b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c6d75b56-hfrpb" Aug 04 10:28:48.782490 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.782237 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cd41804f-6c04-4553-92c5-1d6b48572f5b-hub\") pod \"cluster-proxy-proxy-agent-56c6d75b56-hfrpb\" (UID: \"cd41804f-6c04-4553-92c5-1d6b48572f5b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c6d75b56-hfrpb" Aug 04 10:28:48.782490 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.782260 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cd41804f-6c04-4553-92c5-1d6b48572f5b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-56c6d75b56-hfrpb\" (UID: \"cd41804f-6c04-4553-92c5-1d6b48572f5b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c6d75b56-hfrpb" Aug 04 10:28:48.785858 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.785832 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/165c32fb-4eac-42f0-88de-d6fe0453ad99-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-665fbd9b58-4hzt6\" (UID: \"165c32fb-4eac-42f0-88de-d6fe0453ad99\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-665fbd9b58-4hzt6" Aug 04 10:28:48.795194 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.795164 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8h9k\" (UniqueName: \"kubernetes.io/projected/165c32fb-4eac-42f0-88de-d6fe0453ad99-kube-api-access-g8h9k\") pod \"managed-serviceaccount-addon-agent-665fbd9b58-4hzt6\" (UID: \"165c32fb-4eac-42f0-88de-d6fe0453ad99\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-665fbd9b58-4hzt6" Aug 04 10:28:48.884105 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.883458 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cd41804f-6c04-4553-92c5-1d6b48572f5b-hub\") pod \"cluster-proxy-proxy-agent-56c6d75b56-hfrpb\" (UID: \"cd41804f-6c04-4553-92c5-1d6b48572f5b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c6d75b56-hfrpb" Aug 04 10:28:48.884105 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.883543 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cd41804f-6c04-4553-92c5-1d6b48572f5b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-56c6d75b56-hfrpb\" (UID: \"cd41804f-6c04-4553-92c5-1d6b48572f5b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c6d75b56-hfrpb" Aug 04 10:28:48.884105 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.883596 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd41804f-6c04-4553-92c5-1d6b48572f5b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-56c6d75b56-hfrpb\" (UID: \"cd41804f-6c04-4553-92c5-1d6b48572f5b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c6d75b56-hfrpb" Aug 04 10:28:48.884105 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.883625 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6wmk\" (UniqueName: \"kubernetes.io/projected/cd41804f-6c04-4553-92c5-1d6b48572f5b-kube-api-access-j6wmk\") pod \"cluster-proxy-proxy-agent-56c6d75b56-hfrpb\" (UID: \"cd41804f-6c04-4553-92c5-1d6b48572f5b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c6d75b56-hfrpb" Aug 04 10:28:48.884105 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.883675 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd41804f-6c04-4553-92c5-1d6b48572f5b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-56c6d75b56-hfrpb\" (UID: \"cd41804f-6c04-4553-92c5-1d6b48572f5b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c6d75b56-hfrpb" Aug 04 10:28:48.884105 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.883728 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cd41804f-6c04-4553-92c5-1d6b48572f5b-ca\") pod \"cluster-proxy-proxy-agent-56c6d75b56-hfrpb\" (UID: \"cd41804f-6c04-4553-92c5-1d6b48572f5b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c6d75b56-hfrpb" Aug 04 10:28:48.884806 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.884780 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd41804f-6c04-4553-92c5-1d6b48572f5b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-56c6d75b56-hfrpb\" (UID: \"cd41804f-6c04-4553-92c5-1d6b48572f5b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c6d75b56-hfrpb" Aug 04 10:28:48.889120 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.887507 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd41804f-6c04-4553-92c5-1d6b48572f5b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-56c6d75b56-hfrpb\" (UID: \"cd41804f-6c04-4553-92c5-1d6b48572f5b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c6d75b56-hfrpb" Aug 04 10:28:48.889120 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.888233 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cd41804f-6c04-4553-92c5-1d6b48572f5b-hub\") pod \"cluster-proxy-proxy-agent-56c6d75b56-hfrpb\" (UID: \"cd41804f-6c04-4553-92c5-1d6b48572f5b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c6d75b56-hfrpb" Aug 04 10:28:48.889759 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.889736 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cd41804f-6c04-4553-92c5-1d6b48572f5b-ca\") pod \"cluster-proxy-proxy-agent-56c6d75b56-hfrpb\" (UID: \"cd41804f-6c04-4553-92c5-1d6b48572f5b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c6d75b56-hfrpb" Aug 04 10:28:48.889879 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.889734 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cd41804f-6c04-4553-92c5-1d6b48572f5b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-56c6d75b56-hfrpb\" (UID: \"cd41804f-6c04-4553-92c5-1d6b48572f5b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c6d75b56-hfrpb" Aug 04 10:28:48.908584 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.908529 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6wmk\" (UniqueName: \"kubernetes.io/projected/cd41804f-6c04-4553-92c5-1d6b48572f5b-kube-api-access-j6wmk\") pod \"cluster-proxy-proxy-agent-56c6d75b56-hfrpb\" (UID: \"cd41804f-6c04-4553-92c5-1d6b48572f5b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c6d75b56-hfrpb" Aug 04 10:28:48.975210 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.974923 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-665fbd9b58-4hzt6" Aug 04 10:28:48.983590 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:48.983460 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c6d75b56-hfrpb" Aug 04 10:28:49.206744 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:49.206579 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-665fbd9b58-4hzt6"] Aug 04 10:28:49.216367 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:49.215762 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c6d75b56-hfrpb"] Aug 04 10:28:49.219974 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:28:49.219230 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd41804f_6c04_4553_92c5_1d6b48572f5b.slice/crio-73825f84470b8f830032cc95cb268a7423f8d557f72540f3e59dd8ee907417a4 WatchSource:0}: Error finding container 73825f84470b8f830032cc95cb268a7423f8d557f72540f3e59dd8ee907417a4: Status 404 returned error can't find the container with id 73825f84470b8f830032cc95cb268a7423f8d557f72540f3e59dd8ee907417a4 Aug 04 10:28:49.388413 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:49.387319 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wc2fl" event={"ID":"0db48901-2221-480e-97b0-cd6be997aaef","Type":"ContainerStarted","Data":"f902d91cf1f3d8081abb9c8e37facfa2f19391572930c3a5fc806886d0f3e97b"} Aug 04 10:28:49.388413 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:49.387417 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wc2fl" event={"ID":"0db48901-2221-480e-97b0-cd6be997aaef","Type":"ContainerStarted","Data":"d1ffaf07f7a14ffccc2e44a9dbfbd38e763e3631f9f4dc2ce4d04891e81239bc"} Aug 04 10:28:49.390631 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:49.390510 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c6d75b56-hfrpb" event={"ID":"cd41804f-6c04-4553-92c5-1d6b48572f5b","Type":"ContainerStarted","Data":"73825f84470b8f830032cc95cb268a7423f8d557f72540f3e59dd8ee907417a4"} Aug 04 10:28:49.392644 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:49.392614 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-665fbd9b58-4hzt6" event={"ID":"165c32fb-4eac-42f0-88de-d6fe0453ad99","Type":"ContainerStarted","Data":"e20beb365d60e1704a25bcb2638526811e7acfd806583504fbc3884e55cd1863"} Aug 04 10:28:50.437816 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:50.436927 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wc2fl" podStartSLOduration=67.757737246 podStartE2EDuration="1m9.43690441s" totalImagesPullingTime="1.679167164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:27:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:49.407761612 +0000 UTC m=+68.828649023" watchObservedRunningTime="2026-08-04 10:28:50.43690441 +0000 UTC m=+69.857791823" Aug 04 10:28:54.040871 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:54.040830 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k4lhp\" (UID: \"5c6f919e-0e11-4f1d-a2f0-1db97b80e53d\") " pod="openshift-insights/insights-runtime-extractor-k4lhp" Aug 04 10:28:54.043503 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:54.043481 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c6f919e-0e11-4f1d-a2f0-1db97b80e53d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k4lhp\" (UID: \"5c6f919e-0e11-4f1d-a2f0-1db97b80e53d\") " pod="openshift-insights/insights-runtime-extractor-k4lhp" Aug 04 10:28:54.208866 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:54.208830 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ntlkp\"" Aug 04 10:28:54.216383 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:54.216351 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k4lhp" Aug 04 10:28:56.415448 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:56.415262 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vnz6r" event={"ID":"b8276ab4-da95-49d7-81ed-bbc401ac406a","Type":"ContainerStarted","Data":"ef7ef09f426dc9cc980518f698a49ca375905a2b13b4d6d114df0d263edd6e52"} Aug 04 10:28:56.415957 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:56.415592 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vnz6r" Aug 04 10:28:56.420731 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:56.420300 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c6d75b56-hfrpb" event={"ID":"cd41804f-6c04-4553-92c5-1d6b48572f5b","Type":"ContainerStarted","Data":"38d605b75ac40ee2cdf6c8f56bb27fe66959848f7608620339ce44e2690be8af"} Aug 04 10:28:56.425467 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:56.424764 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k4lhp"] Aug 04 10:28:56.426650 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:56.426127 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-665fbd9b58-4hzt6" event={"ID":"165c32fb-4eac-42f0-88de-d6fe0453ad99","Type":"ContainerStarted","Data":"b63ee9d5dab5d40d597eba3162bafd64cc9825a7e0d36837598d0897dbef0322"} Aug 04 10:28:56.466490 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:56.465726 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vnz6r" podStartSLOduration=66.415181088 podStartE2EDuration="1m15.465680257s" totalImagesPullingTime="9.050499169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:27:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:56.440597882 +0000 UTC m=+75.861485294" watchObservedRunningTime="2026-08-04 10:28:56.465680257 +0000 UTC m=+75.886567716" Aug 04 10:28:56.466723 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:56.466675 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-665fbd9b58-4hzt6" podStartSLOduration=1.434566415 podStartE2EDuration="8.466663813s" totalImagesPullingTime="7.032097398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:28:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:56.464110312 +0000 UTC m=+75.884997724" watchObservedRunningTime="2026-08-04 10:28:56.466663813 +0000 UTC m=+75.887551224" Aug 04 10:28:57.431428 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:57.431379 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c6d75b56-hfrpb" event={"ID":"cd41804f-6c04-4553-92c5-1d6b48572f5b","Type":"ContainerStarted","Data":"ce3f5689604bcb9c043e083b941a63e128d17dd259b3fe69c440ec8d298d6dd7"} Aug 04 10:28:57.431919 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:57.431436 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c6d75b56-hfrpb" event={"ID":"cd41804f-6c04-4553-92c5-1d6b48572f5b","Type":"ContainerStarted","Data":"40e6950b6554dc5a8dab24d3d9e580124831ed88e43f24897b5a4548ab0ff5a5"} Aug 04 10:28:57.432924 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:57.432902 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rzhxt" event={"ID":"b0a98e70-0e23-4348-9b6d-93b4081069df","Type":"ContainerStarted","Data":"588c5921b95ba785f6b55e092747e88d9901a5803cfe4308dde1c3e4a26ed01b"} Aug 04 10:28:57.434809 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:57.434780 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qjvd4" event={"ID":"2d4fd3c4-711b-4896-83a7-18dc02e85c7d","Type":"ContainerStarted","Data":"e86e6cec836e86228d6bb6d133d1b8c110556c427e650d3f55c6bfee1d0992d9"} Aug 04 10:28:57.434923 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:57.434835 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qjvd4" Aug 04 10:28:57.434923 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:57.434853 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qjvd4" event={"ID":"2d4fd3c4-711b-4896-83a7-18dc02e85c7d","Type":"ContainerStarted","Data":"1eed2f42e0acce2129b59486671c04f2233bd010b42b79dbc74f2f5c2680801d"} Aug 04 10:28:57.436581 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:57.436541 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k4lhp" event={"ID":"5c6f919e-0e11-4f1d-a2f0-1db97b80e53d","Type":"ContainerStarted","Data":"e371112a0c60d0e1c5573c7a655174e511f0957de0888195084f007b5a1f64ab"} Aug 04 10:28:57.436667 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:57.436614 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k4lhp" event={"ID":"5c6f919e-0e11-4f1d-a2f0-1db97b80e53d","Type":"ContainerStarted","Data":"5a2361936e34130638d59bd8908c75b30fecaa21cd073e0fd4fe49ee93bbde7e"} Aug 04 10:28:57.436667 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:57.436644 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k4lhp" event={"ID":"5c6f919e-0e11-4f1d-a2f0-1db97b80e53d","Type":"ContainerStarted","Data":"ab6f882754ef5dfe0be6f452ed90e540ab2c7dd3948b69811f84235a22ace32e"} Aug 04 10:28:57.457311 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:57.457249 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c6d75b56-hfrpb" podStartSLOduration=2.362747176 podStartE2EDuration="9.457233865s" totalImagesPullingTime="7.094486689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:28:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:57.456724959 +0000 UTC m=+76.877612369" watchObservedRunningTime="2026-08-04 10:28:57.457233865 +0000 UTC m=+76.878121276" Aug 04 10:28:57.475839 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:57.475772 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rzhxt" podStartSLOduration=67.422855875 podStartE2EDuration="1m16.475752122s" totalImagesPullingTime="9.052896247s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:27:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:57.47359533 +0000 UTC m=+76.894482741" watchObservedRunningTime="2026-08-04 10:28:57.475752122 +0000 UTC m=+76.896639535" Aug 04 10:28:57.493627 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:57.493549 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qjvd4" podStartSLOduration=34.089227395 podStartE2EDuration="42.493532594s" totalImagesPullingTime="8.404305199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:28:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:57.493148592 +0000 UTC m=+76.914036029" watchObservedRunningTime="2026-08-04 10:28:57.493532594 +0000 UTC m=+76.914420005" Aug 04 10:28:59.445991 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:59.445953 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k4lhp" event={"ID":"5c6f919e-0e11-4f1d-a2f0-1db97b80e53d","Type":"ContainerStarted","Data":"5aea9e36588774c83cd5e2a406f238cd17a61f6db6689c06b15d393ab75347c6"} Aug 04 10:28:59.466520 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:28:59.466473 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-k4lhp" podStartSLOduration=35.634967666 podStartE2EDuration="37.466455985s" totalImagesPullingTime="1.831488319s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:28:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:59.466045696 +0000 UTC m=+78.886933109" watchObservedRunningTime="2026-08-04 10:28:59.466455985 +0000 UTC m=+78.887343395" Aug 04 10:29:01.382862 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.382744 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jrmjm"] Aug 04 10:29:01.385822 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.385803 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jrmjm" Aug 04 10:29:01.388797 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.388777 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Aug 04 10:29:01.388893 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.388823 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Aug 04 10:29:01.388893 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.388823 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Aug 04 10:29:01.390154 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.390141 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Aug 04 10:29:01.390154 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.390148 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Aug 04 10:29:01.390246 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.390151 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rg4dt\"" Aug 04 10:29:01.390420 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.390408 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Aug 04 10:29:01.504962 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.504928 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b29e6fab-f600-41b5-a265-d82afbedb3be-node-exporter-tls\") pod \"node-exporter-jrmjm\" (UID: \"b29e6fab-f600-41b5-a265-d82afbedb3be\") " pod="openshift-monitoring/node-exporter-jrmjm" Aug 04 10:29:01.504962 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.504969 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b29e6fab-f600-41b5-a265-d82afbedb3be-sys\") pod \"node-exporter-jrmjm\" (UID: \"b29e6fab-f600-41b5-a265-d82afbedb3be\") " pod="openshift-monitoring/node-exporter-jrmjm" Aug 04 10:29:01.505197 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.504989 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b29e6fab-f600-41b5-a265-d82afbedb3be-node-exporter-accelerators-collector-config\") pod \"node-exporter-jrmjm\" (UID: \"b29e6fab-f600-41b5-a265-d82afbedb3be\") " pod="openshift-monitoring/node-exporter-jrmjm" Aug 04 10:29:01.505197 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.505098 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b29e6fab-f600-41b5-a265-d82afbedb3be-root\") pod \"node-exporter-jrmjm\" (UID: \"b29e6fab-f600-41b5-a265-d82afbedb3be\") " pod="openshift-monitoring/node-exporter-jrmjm" Aug 04 10:29:01.505197 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.505126 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b29e6fab-f600-41b5-a265-d82afbedb3be-node-exporter-textfile\") pod \"node-exporter-jrmjm\" (UID: \"b29e6fab-f600-41b5-a265-d82afbedb3be\") " pod="openshift-monitoring/node-exporter-jrmjm" Aug 04 10:29:01.505197 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.505144 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hxf9\" (UniqueName: \"kubernetes.io/projected/b29e6fab-f600-41b5-a265-d82afbedb3be-kube-api-access-8hxf9\") pod \"node-exporter-jrmjm\" (UID: \"b29e6fab-f600-41b5-a265-d82afbedb3be\") " pod="openshift-monitoring/node-exporter-jrmjm" Aug 04 10:29:01.505197 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.505161 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b29e6fab-f600-41b5-a265-d82afbedb3be-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jrmjm\" (UID: \"b29e6fab-f600-41b5-a265-d82afbedb3be\") " pod="openshift-monitoring/node-exporter-jrmjm" Aug 04 10:29:01.505372 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.505218 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b29e6fab-f600-41b5-a265-d82afbedb3be-metrics-client-ca\") pod \"node-exporter-jrmjm\" (UID: \"b29e6fab-f600-41b5-a265-d82afbedb3be\") " pod="openshift-monitoring/node-exporter-jrmjm" Aug 04 10:29:01.505372 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.505256 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b29e6fab-f600-41b5-a265-d82afbedb3be-node-exporter-wtmp\") pod \"node-exporter-jrmjm\" (UID: \"b29e6fab-f600-41b5-a265-d82afbedb3be\") " pod="openshift-monitoring/node-exporter-jrmjm" Aug 04 10:29:01.606165 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.606135 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b29e6fab-f600-41b5-a265-d82afbedb3be-sys\") pod \"node-exporter-jrmjm\" (UID: \"b29e6fab-f600-41b5-a265-d82afbedb3be\") " pod="openshift-monitoring/node-exporter-jrmjm" Aug 04 10:29:01.606165 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.606169 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b29e6fab-f600-41b5-a265-d82afbedb3be-node-exporter-accelerators-collector-config\") pod \"node-exporter-jrmjm\" (UID: \"b29e6fab-f600-41b5-a265-d82afbedb3be\") " pod="openshift-monitoring/node-exporter-jrmjm" Aug 04 10:29:01.606380 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.606200 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b29e6fab-f600-41b5-a265-d82afbedb3be-root\") pod \"node-exporter-jrmjm\" (UID: \"b29e6fab-f600-41b5-a265-d82afbedb3be\") " pod="openshift-monitoring/node-exporter-jrmjm" Aug 04 10:29:01.606380 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.606225 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b29e6fab-f600-41b5-a265-d82afbedb3be-node-exporter-textfile\") pod \"node-exporter-jrmjm\" (UID: \"b29e6fab-f600-41b5-a265-d82afbedb3be\") " pod="openshift-monitoring/node-exporter-jrmjm" Aug 04 10:29:01.606380 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.606248 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hxf9\" (UniqueName: \"kubernetes.io/projected/b29e6fab-f600-41b5-a265-d82afbedb3be-kube-api-access-8hxf9\") pod \"node-exporter-jrmjm\" (UID: \"b29e6fab-f600-41b5-a265-d82afbedb3be\") " pod="openshift-monitoring/node-exporter-jrmjm" Aug 04 10:29:01.606380 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.606252 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b29e6fab-f600-41b5-a265-d82afbedb3be-sys\") pod \"node-exporter-jrmjm\" (UID: \"b29e6fab-f600-41b5-a265-d82afbedb3be\") " pod="openshift-monitoring/node-exporter-jrmjm" Aug 04 10:29:01.606380 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.606273 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b29e6fab-f600-41b5-a265-d82afbedb3be-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jrmjm\" (UID: \"b29e6fab-f600-41b5-a265-d82afbedb3be\") " pod="openshift-monitoring/node-exporter-jrmjm" Aug 04 10:29:01.606380 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.606303 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b29e6fab-f600-41b5-a265-d82afbedb3be-metrics-client-ca\") pod \"node-exporter-jrmjm\" (UID: \"b29e6fab-f600-41b5-a265-d82afbedb3be\") " pod="openshift-monitoring/node-exporter-jrmjm" Aug 04 10:29:01.606380 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.606330 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b29e6fab-f600-41b5-a265-d82afbedb3be-node-exporter-wtmp\") pod \"node-exporter-jrmjm\" (UID: \"b29e6fab-f600-41b5-a265-d82afbedb3be\") " pod="openshift-monitoring/node-exporter-jrmjm" Aug 04 10:29:01.606380 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.606355 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b29e6fab-f600-41b5-a265-d82afbedb3be-root\") pod \"node-exporter-jrmjm\" (UID: \"b29e6fab-f600-41b5-a265-d82afbedb3be\") " pod="openshift-monitoring/node-exporter-jrmjm" Aug 04 10:29:01.606751 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.606366 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b29e6fab-f600-41b5-a265-d82afbedb3be-node-exporter-tls\") pod \"node-exporter-jrmjm\" (UID: \"b29e6fab-f600-41b5-a265-d82afbedb3be\") " pod="openshift-monitoring/node-exporter-jrmjm" Aug 04 10:29:01.606751 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.606589 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b29e6fab-f600-41b5-a265-d82afbedb3be-node-exporter-wtmp\") pod \"node-exporter-jrmjm\" (UID: \"b29e6fab-f600-41b5-a265-d82afbedb3be\") " pod="openshift-monitoring/node-exporter-jrmjm" Aug 04 10:29:01.606751 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.606704 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b29e6fab-f600-41b5-a265-d82afbedb3be-node-exporter-textfile\") pod \"node-exporter-jrmjm\" (UID: \"b29e6fab-f600-41b5-a265-d82afbedb3be\") " pod="openshift-monitoring/node-exporter-jrmjm" Aug 04 10:29:01.606894 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.606879 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b29e6fab-f600-41b5-a265-d82afbedb3be-metrics-client-ca\") pod \"node-exporter-jrmjm\" (UID: \"b29e6fab-f600-41b5-a265-d82afbedb3be\") " pod="openshift-monitoring/node-exporter-jrmjm" Aug 04 10:29:01.606944 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.606879 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b29e6fab-f600-41b5-a265-d82afbedb3be-node-exporter-accelerators-collector-config\") pod \"node-exporter-jrmjm\" (UID: \"b29e6fab-f600-41b5-a265-d82afbedb3be\") " pod="openshift-monitoring/node-exporter-jrmjm" Aug 04 10:29:01.609008 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.608982 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b29e6fab-f600-41b5-a265-d82afbedb3be-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jrmjm\" (UID: \"b29e6fab-f600-41b5-a265-d82afbedb3be\") " pod="openshift-monitoring/node-exporter-jrmjm" Aug 04 10:29:01.609133 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.609059 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b29e6fab-f600-41b5-a265-d82afbedb3be-node-exporter-tls\") pod \"node-exporter-jrmjm\" (UID: \"b29e6fab-f600-41b5-a265-d82afbedb3be\") " pod="openshift-monitoring/node-exporter-jrmjm" Aug 04 10:29:01.615402 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.615372 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hxf9\" (UniqueName: \"kubernetes.io/projected/b29e6fab-f600-41b5-a265-d82afbedb3be-kube-api-access-8hxf9\") pod \"node-exporter-jrmjm\" (UID: \"b29e6fab-f600-41b5-a265-d82afbedb3be\") " pod="openshift-monitoring/node-exporter-jrmjm" Aug 04 10:29:01.694407 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:01.694322 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jrmjm" Aug 04 10:29:01.705894 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:29:01.705863 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb29e6fab_f600_41b5_a265_d82afbedb3be.slice/crio-f8087d0c3fd42783ee7795c8727cc22c38ebb50ccd73f4a915b2b075309240c0 WatchSource:0}: Error finding container f8087d0c3fd42783ee7795c8727cc22c38ebb50ccd73f4a915b2b075309240c0: Status 404 returned error can't find the container with id f8087d0c3fd42783ee7795c8727cc22c38ebb50ccd73f4a915b2b075309240c0 Aug 04 10:29:02.460167 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:02.460117 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jrmjm" event={"ID":"b29e6fab-f600-41b5-a265-d82afbedb3be","Type":"ContainerStarted","Data":"f8087d0c3fd42783ee7795c8727cc22c38ebb50ccd73f4a915b2b075309240c0"} Aug 04 10:29:03.464881 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:03.464845 2580 generic.go:356] "Generic (PLEG): container finished" podID="b29e6fab-f600-41b5-a265-d82afbedb3be" containerID="4452953d7aaff2ea8c1d4a94bf0ceb3a2a5079c5bfe05bdf6f7815cde91c0bca" exitCode=0 Aug 04 10:29:03.465264 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:03.464908 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jrmjm" event={"ID":"b29e6fab-f600-41b5-a265-d82afbedb3be","Type":"ContainerDied","Data":"4452953d7aaff2ea8c1d4a94bf0ceb3a2a5079c5bfe05bdf6f7815cde91c0bca"} Aug 04 10:29:04.469955 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:04.469920 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jrmjm" event={"ID":"b29e6fab-f600-41b5-a265-d82afbedb3be","Type":"ContainerStarted","Data":"6d35aedd5619f6f667412fc1bbc6797495647aba2283d567519c67d7753e91ad"} Aug 04 10:29:04.470333 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:04.469972 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jrmjm" event={"ID":"b29e6fab-f600-41b5-a265-d82afbedb3be","Type":"ContainerStarted","Data":"0463b0d5e50a59001ae5a8bec056f129de96ea7802b69e0e4f7fc513b03cd32d"} Aug 04 10:29:04.494812 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:04.494756 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jrmjm" podStartSLOduration=2.194696606 podStartE2EDuration="3.494737206s" totalImagesPullingTime="1.226785353s" totalInitContainerRuntime="73.255247ms" isStatefulPod=true podCreationTimestamp="2026-08-04 10:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:29:04.494017219 +0000 UTC m=+83.914904629" watchObservedRunningTime="2026-08-04 10:29:04.494737206 +0000 UTC m=+83.915624653" Aug 04 10:29:05.697162 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:05.697132 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-85799b647-lxhxn"] Aug 04 10:29:05.700148 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:05.700134 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-85799b647-lxhxn" Aug 04 10:29:05.703391 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:05.703363 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Aug 04 10:29:05.703508 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:05.703465 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Aug 04 10:29:05.703715 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:05.703699 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-wf7qx\"" Aug 04 10:29:05.704976 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:05.704958 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Aug 04 10:29:05.705129 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:05.704967 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8g8guobjgg7dg\"" Aug 04 10:29:05.705129 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:05.705079 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Aug 04 10:29:05.719700 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:05.719674 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-85799b647-lxhxn"] Aug 04 10:29:05.833670 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:05.833637 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/623b61a0-e92d-48c6-96db-e48c57399f90-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-85799b647-lxhxn\" (UID: \"623b61a0-e92d-48c6-96db-e48c57399f90\") " pod="openshift-monitoring/metrics-server-85799b647-lxhxn" Aug 04 10:29:05.833884 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:05.833679 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/623b61a0-e92d-48c6-96db-e48c57399f90-metrics-server-audit-profiles\") pod \"metrics-server-85799b647-lxhxn\" (UID: \"623b61a0-e92d-48c6-96db-e48c57399f90\") " pod="openshift-monitoring/metrics-server-85799b647-lxhxn" Aug 04 10:29:05.833884 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:05.833718 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/623b61a0-e92d-48c6-96db-e48c57399f90-client-ca-bundle\") pod \"metrics-server-85799b647-lxhxn\" (UID: \"623b61a0-e92d-48c6-96db-e48c57399f90\") " pod="openshift-monitoring/metrics-server-85799b647-lxhxn" Aug 04 10:29:05.833884 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:05.833745 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/623b61a0-e92d-48c6-96db-e48c57399f90-secret-metrics-server-client-certs\") pod \"metrics-server-85799b647-lxhxn\" (UID: \"623b61a0-e92d-48c6-96db-e48c57399f90\") " pod="openshift-monitoring/metrics-server-85799b647-lxhxn" Aug 04 10:29:05.833884 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:05.833777 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/623b61a0-e92d-48c6-96db-e48c57399f90-audit-log\") pod \"metrics-server-85799b647-lxhxn\" (UID: \"623b61a0-e92d-48c6-96db-e48c57399f90\") " pod="openshift-monitoring/metrics-server-85799b647-lxhxn" Aug 04 10:29:05.833884 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:05.833810 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j97m5\" (UniqueName: \"kubernetes.io/projected/623b61a0-e92d-48c6-96db-e48c57399f90-kube-api-access-j97m5\") pod \"metrics-server-85799b647-lxhxn\" (UID: \"623b61a0-e92d-48c6-96db-e48c57399f90\") " pod="openshift-monitoring/metrics-server-85799b647-lxhxn" Aug 04 10:29:05.833884 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:05.833881 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/623b61a0-e92d-48c6-96db-e48c57399f90-secret-metrics-server-tls\") pod \"metrics-server-85799b647-lxhxn\" (UID: \"623b61a0-e92d-48c6-96db-e48c57399f90\") " pod="openshift-monitoring/metrics-server-85799b647-lxhxn" Aug 04 10:29:05.934408 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:05.934371 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/623b61a0-e92d-48c6-96db-e48c57399f90-secret-metrics-server-tls\") pod \"metrics-server-85799b647-lxhxn\" (UID: \"623b61a0-e92d-48c6-96db-e48c57399f90\") " pod="openshift-monitoring/metrics-server-85799b647-lxhxn" Aug 04 10:29:05.934408 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:05.934410 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/623b61a0-e92d-48c6-96db-e48c57399f90-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-85799b647-lxhxn\" (UID: \"623b61a0-e92d-48c6-96db-e48c57399f90\") " pod="openshift-monitoring/metrics-server-85799b647-lxhxn" Aug 04 10:29:05.934677 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:05.934432 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/623b61a0-e92d-48c6-96db-e48c57399f90-metrics-server-audit-profiles\") pod \"metrics-server-85799b647-lxhxn\" (UID: \"623b61a0-e92d-48c6-96db-e48c57399f90\") " pod="openshift-monitoring/metrics-server-85799b647-lxhxn" Aug 04 10:29:05.934677 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:05.934453 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/623b61a0-e92d-48c6-96db-e48c57399f90-client-ca-bundle\") pod \"metrics-server-85799b647-lxhxn\" (UID: \"623b61a0-e92d-48c6-96db-e48c57399f90\") " pod="openshift-monitoring/metrics-server-85799b647-lxhxn" Aug 04 10:29:05.934677 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:05.934472 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/623b61a0-e92d-48c6-96db-e48c57399f90-secret-metrics-server-client-certs\") pod \"metrics-server-85799b647-lxhxn\" (UID: \"623b61a0-e92d-48c6-96db-e48c57399f90\") " pod="openshift-monitoring/metrics-server-85799b647-lxhxn" Aug 04 10:29:05.934677 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:05.934487 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/623b61a0-e92d-48c6-96db-e48c57399f90-audit-log\") pod \"metrics-server-85799b647-lxhxn\" (UID: \"623b61a0-e92d-48c6-96db-e48c57399f90\") " pod="openshift-monitoring/metrics-server-85799b647-lxhxn" Aug 04 10:29:05.934677 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:05.934505 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j97m5\" (UniqueName: \"kubernetes.io/projected/623b61a0-e92d-48c6-96db-e48c57399f90-kube-api-access-j97m5\") pod \"metrics-server-85799b647-lxhxn\" (UID: \"623b61a0-e92d-48c6-96db-e48c57399f90\") " pod="openshift-monitoring/metrics-server-85799b647-lxhxn" Aug 04 10:29:05.935000 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:05.934968 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/623b61a0-e92d-48c6-96db-e48c57399f90-audit-log\") pod \"metrics-server-85799b647-lxhxn\" (UID: \"623b61a0-e92d-48c6-96db-e48c57399f90\") " pod="openshift-monitoring/metrics-server-85799b647-lxhxn" Aug 04 10:29:05.935482 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:05.935457 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/623b61a0-e92d-48c6-96db-e48c57399f90-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-85799b647-lxhxn\" (UID: \"623b61a0-e92d-48c6-96db-e48c57399f90\") " pod="openshift-monitoring/metrics-server-85799b647-lxhxn" Aug 04 10:29:05.935753 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:05.935733 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/623b61a0-e92d-48c6-96db-e48c57399f90-metrics-server-audit-profiles\") pod \"metrics-server-85799b647-lxhxn\" (UID: \"623b61a0-e92d-48c6-96db-e48c57399f90\") " pod="openshift-monitoring/metrics-server-85799b647-lxhxn" Aug 04 10:29:05.937143 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:05.937119 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/623b61a0-e92d-48c6-96db-e48c57399f90-secret-metrics-server-client-certs\") pod \"metrics-server-85799b647-lxhxn\" (UID: \"623b61a0-e92d-48c6-96db-e48c57399f90\") " pod="openshift-monitoring/metrics-server-85799b647-lxhxn" Aug 04 10:29:05.937143 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:05.937134 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/623b61a0-e92d-48c6-96db-e48c57399f90-secret-metrics-server-tls\") pod \"metrics-server-85799b647-lxhxn\" (UID: \"623b61a0-e92d-48c6-96db-e48c57399f90\") " pod="openshift-monitoring/metrics-server-85799b647-lxhxn" Aug 04 10:29:05.937240 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:05.937187 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/623b61a0-e92d-48c6-96db-e48c57399f90-client-ca-bundle\") pod \"metrics-server-85799b647-lxhxn\" (UID: \"623b61a0-e92d-48c6-96db-e48c57399f90\") " pod="openshift-monitoring/metrics-server-85799b647-lxhxn" Aug 04 10:29:05.944241 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:05.944216 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j97m5\" (UniqueName: \"kubernetes.io/projected/623b61a0-e92d-48c6-96db-e48c57399f90-kube-api-access-j97m5\") pod \"metrics-server-85799b647-lxhxn\" (UID: \"623b61a0-e92d-48c6-96db-e48c57399f90\") " pod="openshift-monitoring/metrics-server-85799b647-lxhxn" Aug 04 10:29:06.009401 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:06.009282 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-85799b647-lxhxn" Aug 04 10:29:06.145087 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:06.145054 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-85799b647-lxhxn"] Aug 04 10:29:06.148639 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:29:06.148614 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod623b61a0_e92d_48c6_96db_e48c57399f90.slice/crio-0f8508dd90131401bd56c3332e06778d03254bd4e16980ded67431a856452953 WatchSource:0}: Error finding container 0f8508dd90131401bd56c3332e06778d03254bd4e16980ded67431a856452953: Status 404 returned error can't find the container with id 0f8508dd90131401bd56c3332e06778d03254bd4e16980ded67431a856452953 Aug 04 10:29:06.477993 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:06.477949 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-85799b647-lxhxn" event={"ID":"623b61a0-e92d-48c6-96db-e48c57399f90","Type":"ContainerStarted","Data":"0f8508dd90131401bd56c3332e06778d03254bd4e16980ded67431a856452953"} Aug 04 10:29:07.443584 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:07.443533 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qjvd4" Aug 04 10:29:08.487781 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:08.487737 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-85799b647-lxhxn" event={"ID":"623b61a0-e92d-48c6-96db-e48c57399f90","Type":"ContainerStarted","Data":"ced438c04107b238e67e04a0ef20c6d82ce163f11ec78024ff23800a7e358c7b"} Aug 04 10:29:08.515436 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:08.515389 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-85799b647-lxhxn" podStartSLOduration=1.995133097 podStartE2EDuration="3.515372315s" totalImagesPullingTime="1.520239218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:29:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:29:08.513757342 +0000 UTC m=+87.934644753" watchObservedRunningTime="2026-08-04 10:29:08.515372315 +0000 UTC m=+87.936259726" Aug 04 10:29:10.402013 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:10.401985 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:29:15.439635 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:15.439587 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" podUID="80278eaf-3e7b-4e5b-a540-8314ecd09e96" containerName="registry" containerID="cri-o://076fb34a4b23c6e77e0d4c1aafec6fd95b996594c61088901b35a55875aebf1d" gracePeriod=30 Aug 04 10:29:15.670894 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:15.670870 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:29:15.810186 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:15.810092 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-registry-tls\") pod \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " Aug 04 10:29:15.810186 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:15.810146 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80278eaf-3e7b-4e5b-a540-8314ecd09e96-trusted-ca\") pod \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " Aug 04 10:29:15.810186 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:15.810187 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/80278eaf-3e7b-4e5b-a540-8314ecd09e96-image-registry-private-configuration\") pod \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " Aug 04 10:29:15.810446 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:15.810216 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/80278eaf-3e7b-4e5b-a540-8314ecd09e96-registry-certificates\") pod \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " Aug 04 10:29:15.810446 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:15.810239 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-bound-sa-token\") pod \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " Aug 04 10:29:15.810446 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:15.810258 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nppgp\" (UniqueName: \"kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-kube-api-access-nppgp\") pod \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " Aug 04 10:29:15.810446 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:15.810275 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/80278eaf-3e7b-4e5b-a540-8314ecd09e96-installation-pull-secrets\") pod \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " Aug 04 10:29:15.810446 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:15.810308 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/80278eaf-3e7b-4e5b-a540-8314ecd09e96-ca-trust-extracted\") pod \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\" (UID: \"80278eaf-3e7b-4e5b-a540-8314ecd09e96\") " Aug 04 10:29:15.810865 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:15.810714 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80278eaf-3e7b-4e5b-a540-8314ecd09e96-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "80278eaf-3e7b-4e5b-a540-8314ecd09e96" (UID: "80278eaf-3e7b-4e5b-a540-8314ecd09e96"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 10:29:15.810865 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:15.810849 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80278eaf-3e7b-4e5b-a540-8314ecd09e96-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "80278eaf-3e7b-4e5b-a540-8314ecd09e96" (UID: "80278eaf-3e7b-4e5b-a540-8314ecd09e96"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 10:29:15.813031 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:15.812982 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80278eaf-3e7b-4e5b-a540-8314ecd09e96-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "80278eaf-3e7b-4e5b-a540-8314ecd09e96" (UID: "80278eaf-3e7b-4e5b-a540-8314ecd09e96"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 10:29:15.813157 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:15.813041 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-kube-api-access-nppgp" (OuterVolumeSpecName: "kube-api-access-nppgp") pod "80278eaf-3e7b-4e5b-a540-8314ecd09e96" (UID: "80278eaf-3e7b-4e5b-a540-8314ecd09e96"). InnerVolumeSpecName "kube-api-access-nppgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:29:15.813254 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:15.813208 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "80278eaf-3e7b-4e5b-a540-8314ecd09e96" (UID: "80278eaf-3e7b-4e5b-a540-8314ecd09e96"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:29:15.813311 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:15.813287 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "80278eaf-3e7b-4e5b-a540-8314ecd09e96" (UID: "80278eaf-3e7b-4e5b-a540-8314ecd09e96"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:29:15.813412 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:15.813393 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80278eaf-3e7b-4e5b-a540-8314ecd09e96-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "80278eaf-3e7b-4e5b-a540-8314ecd09e96" (UID: "80278eaf-3e7b-4e5b-a540-8314ecd09e96"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 10:29:15.822211 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:15.822183 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80278eaf-3e7b-4e5b-a540-8314ecd09e96-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "80278eaf-3e7b-4e5b-a540-8314ecd09e96" (UID: "80278eaf-3e7b-4e5b-a540-8314ecd09e96"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:29:15.911202 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:15.911158 2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/80278eaf-3e7b-4e5b-a540-8314ecd09e96-image-registry-private-configuration\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:29:15.911202 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:15.911199 2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/80278eaf-3e7b-4e5b-a540-8314ecd09e96-registry-certificates\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:29:15.911202 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:15.911212 2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-bound-sa-token\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:29:15.911202 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:15.911221 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nppgp\" (UniqueName: \"kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-kube-api-access-nppgp\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:29:15.911456 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:15.911229 2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/80278eaf-3e7b-4e5b-a540-8314ecd09e96-installation-pull-secrets\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:29:15.911456 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:15.911239 2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/80278eaf-3e7b-4e5b-a540-8314ecd09e96-ca-trust-extracted\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:29:15.911456 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:15.911247 2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80278eaf-3e7b-4e5b-a540-8314ecd09e96-registry-tls\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:29:15.911456 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:15.911256 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80278eaf-3e7b-4e5b-a540-8314ecd09e96-trusted-ca\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:29:16.517199 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:16.517161 2580 generic.go:356] "Generic (PLEG): container finished" podID="80278eaf-3e7b-4e5b-a540-8314ecd09e96" containerID="076fb34a4b23c6e77e0d4c1aafec6fd95b996594c61088901b35a55875aebf1d" exitCode=0 Aug 04 10:29:16.517676 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:16.517209 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" Aug 04 10:29:16.517676 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:16.517207 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" event={"ID":"80278eaf-3e7b-4e5b-a540-8314ecd09e96","Type":"ContainerDied","Data":"076fb34a4b23c6e77e0d4c1aafec6fd95b996594c61088901b35a55875aebf1d"} Aug 04 10:29:16.517676 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:16.517317 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-597ff7459b-kwzrb" event={"ID":"80278eaf-3e7b-4e5b-a540-8314ecd09e96","Type":"ContainerDied","Data":"fdfec6049eee8f6faca7bb365a9988bf664edcfa9ec962c22b32d92dd9670c91"} Aug 04 10:29:16.517676 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:16.517350 2580 scope.go:117] "RemoveContainer" containerID="076fb34a4b23c6e77e0d4c1aafec6fd95b996594c61088901b35a55875aebf1d" Aug 04 10:29:16.525768 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:16.525750 2580 scope.go:117] "RemoveContainer" containerID="076fb34a4b23c6e77e0d4c1aafec6fd95b996594c61088901b35a55875aebf1d" Aug 04 10:29:16.526051 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:29:16.526032 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"076fb34a4b23c6e77e0d4c1aafec6fd95b996594c61088901b35a55875aebf1d\": container with ID starting with 076fb34a4b23c6e77e0d4c1aafec6fd95b996594c61088901b35a55875aebf1d not found: ID does not exist" containerID="076fb34a4b23c6e77e0d4c1aafec6fd95b996594c61088901b35a55875aebf1d" Aug 04 10:29:16.526112 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:16.526062 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"076fb34a4b23c6e77e0d4c1aafec6fd95b996594c61088901b35a55875aebf1d"} err="failed to get container status \"076fb34a4b23c6e77e0d4c1aafec6fd95b996594c61088901b35a55875aebf1d\": rpc error: code = NotFound desc = could not find container \"076fb34a4b23c6e77e0d4c1aafec6fd95b996594c61088901b35a55875aebf1d\": container with ID starting with 076fb34a4b23c6e77e0d4c1aafec6fd95b996594c61088901b35a55875aebf1d not found: ID does not exist" Aug 04 10:29:16.538199 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:16.538174 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-597ff7459b-kwzrb"] Aug 04 10:29:16.542740 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:16.542719 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-597ff7459b-kwzrb"] Aug 04 10:29:17.097707 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:17.097676 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80278eaf-3e7b-4e5b-a540-8314ecd09e96" path="/var/lib/kubelet/pods/80278eaf-3e7b-4e5b-a540-8314ecd09e96/volumes" Aug 04 10:29:20.320604 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.318876 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5df8f4b68b-4wtbz"] Aug 04 10:29:20.320604 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.319336 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80278eaf-3e7b-4e5b-a540-8314ecd09e96" containerName="registry" Aug 04 10:29:20.320604 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.319352 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="80278eaf-3e7b-4e5b-a540-8314ecd09e96" containerName="registry" Aug 04 10:29:20.320604 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.319480 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="80278eaf-3e7b-4e5b-a540-8314ecd09e96" containerName="registry" Aug 04 10:29:20.324857 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.324822 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5df8f4b68b-4wtbz" Aug 04 10:29:20.328149 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.328128 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Aug 04 10:29:20.329465 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.329446 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Aug 04 10:29:20.329548 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.329503 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Aug 04 10:29:20.329636 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.329553 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Aug 04 10:29:20.329636 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.329615 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Aug 04 10:29:20.329773 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.329674 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Aug 04 10:29:20.329773 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.329696 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Aug 04 10:29:20.329773 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.329744 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-r84x5\"" Aug 04 10:29:20.334184 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.334166 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5df8f4b68b-4wtbz"] Aug 04 10:29:20.334391 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.334376 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Aug 04 10:29:20.441437 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.441401 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b6d5797-cca7-4551-9caf-bf984e82a0c3-service-ca\") pod \"console-5df8f4b68b-4wtbz\" (UID: \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\") " pod="openshift-console/console-5df8f4b68b-4wtbz" Aug 04 10:29:20.441437 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.441438 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b6d5797-cca7-4551-9caf-bf984e82a0c3-trusted-ca-bundle\") pod \"console-5df8f4b68b-4wtbz\" (UID: \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\") " pod="openshift-console/console-5df8f4b68b-4wtbz" Aug 04 10:29:20.441665 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.441462 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b6d5797-cca7-4551-9caf-bf984e82a0c3-console-oauth-config\") pod \"console-5df8f4b68b-4wtbz\" (UID: \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\") " pod="openshift-console/console-5df8f4b68b-4wtbz" Aug 04 10:29:20.441665 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.441536 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4hws\" (UniqueName: \"kubernetes.io/projected/2b6d5797-cca7-4551-9caf-bf984e82a0c3-kube-api-access-t4hws\") pod \"console-5df8f4b68b-4wtbz\" (UID: \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\") " pod="openshift-console/console-5df8f4b68b-4wtbz" Aug 04 10:29:20.441665 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.441615 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b6d5797-cca7-4551-9caf-bf984e82a0c3-oauth-serving-cert\") pod \"console-5df8f4b68b-4wtbz\" (UID: \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\") " pod="openshift-console/console-5df8f4b68b-4wtbz" Aug 04 10:29:20.441665 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.441641 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b6d5797-cca7-4551-9caf-bf984e82a0c3-console-config\") pod \"console-5df8f4b68b-4wtbz\" (UID: \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\") " pod="openshift-console/console-5df8f4b68b-4wtbz" Aug 04 10:29:20.441793 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.441678 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b6d5797-cca7-4551-9caf-bf984e82a0c3-console-serving-cert\") pod \"console-5df8f4b68b-4wtbz\" (UID: \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\") " pod="openshift-console/console-5df8f4b68b-4wtbz" Aug 04 10:29:20.543068 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.543033 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b6d5797-cca7-4551-9caf-bf984e82a0c3-oauth-serving-cert\") pod \"console-5df8f4b68b-4wtbz\" (UID: \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\") " pod="openshift-console/console-5df8f4b68b-4wtbz" Aug 04 10:29:20.543068 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.543072 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b6d5797-cca7-4551-9caf-bf984e82a0c3-console-config\") pod \"console-5df8f4b68b-4wtbz\" (UID: \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\") " pod="openshift-console/console-5df8f4b68b-4wtbz" Aug 04 10:29:20.543354 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.543096 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b6d5797-cca7-4551-9caf-bf984e82a0c3-console-serving-cert\") pod \"console-5df8f4b68b-4wtbz\" (UID: \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\") " pod="openshift-console/console-5df8f4b68b-4wtbz" Aug 04 10:29:20.543354 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.543119 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b6d5797-cca7-4551-9caf-bf984e82a0c3-service-ca\") pod \"console-5df8f4b68b-4wtbz\" (UID: \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\") " pod="openshift-console/console-5df8f4b68b-4wtbz" Aug 04 10:29:20.543354 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.543146 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b6d5797-cca7-4551-9caf-bf984e82a0c3-trusted-ca-bundle\") pod \"console-5df8f4b68b-4wtbz\" (UID: \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\") " pod="openshift-console/console-5df8f4b68b-4wtbz" Aug 04 10:29:20.543354 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.543272 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b6d5797-cca7-4551-9caf-bf984e82a0c3-console-oauth-config\") pod \"console-5df8f4b68b-4wtbz\" (UID: \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\") " pod="openshift-console/console-5df8f4b68b-4wtbz" Aug 04 10:29:20.543354 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.543324 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4hws\" (UniqueName: \"kubernetes.io/projected/2b6d5797-cca7-4551-9caf-bf984e82a0c3-kube-api-access-t4hws\") pod \"console-5df8f4b68b-4wtbz\" (UID: \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\") " pod="openshift-console/console-5df8f4b68b-4wtbz" Aug 04 10:29:20.543918 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.543896 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b6d5797-cca7-4551-9caf-bf984e82a0c3-oauth-serving-cert\") pod \"console-5df8f4b68b-4wtbz\" (UID: \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\") " pod="openshift-console/console-5df8f4b68b-4wtbz" Aug 04 10:29:20.544015 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.543904 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b6d5797-cca7-4551-9caf-bf984e82a0c3-console-config\") pod \"console-5df8f4b68b-4wtbz\" (UID: \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\") " pod="openshift-console/console-5df8f4b68b-4wtbz" Aug 04 10:29:20.544015 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.543959 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b6d5797-cca7-4551-9caf-bf984e82a0c3-service-ca\") pod \"console-5df8f4b68b-4wtbz\" (UID: \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\") " pod="openshift-console/console-5df8f4b68b-4wtbz" Aug 04 10:29:20.544382 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.544361 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b6d5797-cca7-4551-9caf-bf984e82a0c3-trusted-ca-bundle\") pod \"console-5df8f4b68b-4wtbz\" (UID: \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\") " pod="openshift-console/console-5df8f4b68b-4wtbz" Aug 04 10:29:20.545993 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.545972 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b6d5797-cca7-4551-9caf-bf984e82a0c3-console-serving-cert\") pod \"console-5df8f4b68b-4wtbz\" (UID: \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\") " pod="openshift-console/console-5df8f4b68b-4wtbz" Aug 04 10:29:20.545993 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.545982 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b6d5797-cca7-4551-9caf-bf984e82a0c3-console-oauth-config\") pod \"console-5df8f4b68b-4wtbz\" (UID: \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\") " pod="openshift-console/console-5df8f4b68b-4wtbz" Aug 04 10:29:20.555477 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.555442 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4hws\" (UniqueName: \"kubernetes.io/projected/2b6d5797-cca7-4551-9caf-bf984e82a0c3-kube-api-access-t4hws\") pod \"console-5df8f4b68b-4wtbz\" (UID: \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\") " pod="openshift-console/console-5df8f4b68b-4wtbz" Aug 04 10:29:20.633885 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.633853 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5df8f4b68b-4wtbz" Aug 04 10:29:20.768131 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:20.768100 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5df8f4b68b-4wtbz"] Aug 04 10:29:20.771052 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:29:20.771021 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6d5797_cca7_4551_9caf_bf984e82a0c3.slice/crio-74649c95ea5988796788d5150d367a66ce51ad4303b92cbba6836fc321660bbe WatchSource:0}: Error finding container 74649c95ea5988796788d5150d367a66ce51ad4303b92cbba6836fc321660bbe: Status 404 returned error can't find the container with id 74649c95ea5988796788d5150d367a66ce51ad4303b92cbba6836fc321660bbe Aug 04 10:29:21.536873 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:21.536831 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5df8f4b68b-4wtbz" event={"ID":"2b6d5797-cca7-4551-9caf-bf984e82a0c3","Type":"ContainerStarted","Data":"74649c95ea5988796788d5150d367a66ce51ad4303b92cbba6836fc321660bbe"} Aug 04 10:29:24.550600 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:24.550545 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5df8f4b68b-4wtbz" event={"ID":"2b6d5797-cca7-4551-9caf-bf984e82a0c3","Type":"ContainerStarted","Data":"f0fc32af417ed4f3cdc1a8f9d12ee1f5eb8fc5eda54197f3a46e18b31895ad55"} Aug 04 10:29:24.573262 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:24.573207 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5df8f4b68b-4wtbz" podStartSLOduration=1.528616757 podStartE2EDuration="4.573190167s" totalImagesPullingTime="3.04457341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:29:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:29:24.572424422 +0000 UTC m=+103.993311834" watchObservedRunningTime="2026-08-04 10:29:24.573190167 +0000 UTC m=+103.994077578" Aug 04 10:29:26.009459 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:26.009420 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-85799b647-lxhxn" Aug 04 10:29:26.009896 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:26.009487 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-85799b647-lxhxn" Aug 04 10:29:27.438986 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:27.438958 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vnz6r" Aug 04 10:29:30.634850 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:30.634814 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5df8f4b68b-4wtbz" Aug 04 10:29:30.635238 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:30.634878 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5df8f4b68b-4wtbz" Aug 04 10:29:30.639832 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:30.639807 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5df8f4b68b-4wtbz" Aug 04 10:29:31.576643 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:31.576612 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5df8f4b68b-4wtbz" Aug 04 10:29:37.198745 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:37.198715 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qbft2_6ecb558b-6b0a-42a7-a5ae-69fad1241694/serve-healthcheck-canary/0.log" Aug 04 10:29:46.015222 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:46.015187 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-85799b647-lxhxn" Aug 04 10:29:46.019172 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:29:46.019149 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-85799b647-lxhxn" Aug 04 10:31:05.659741 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:05.659706 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5df8f4b68b-4wtbz"] Aug 04 10:31:30.679209 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:30.679171 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5df8f4b68b-4wtbz" podUID="2b6d5797-cca7-4551-9caf-bf984e82a0c3" containerName="console" containerID="cri-o://f0fc32af417ed4f3cdc1a8f9d12ee1f5eb8fc5eda54197f3a46e18b31895ad55" gracePeriod=15 Aug 04 10:31:30.921954 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:30.921927 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5df8f4b68b-4wtbz" Aug 04 10:31:30.967682 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:30.967589 2580 generic.go:356] "Generic (PLEG): container finished" podID="2b6d5797-cca7-4551-9caf-bf984e82a0c3" containerID="f0fc32af417ed4f3cdc1a8f9d12ee1f5eb8fc5eda54197f3a46e18b31895ad55" exitCode=0 Aug 04 10:31:30.967682 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:30.967639 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5df8f4b68b-4wtbz" Aug 04 10:31:30.967851 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:30.967677 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5df8f4b68b-4wtbz" event={"ID":"2b6d5797-cca7-4551-9caf-bf984e82a0c3","Type":"ContainerDied","Data":"f0fc32af417ed4f3cdc1a8f9d12ee1f5eb8fc5eda54197f3a46e18b31895ad55"} Aug 04 10:31:30.967851 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:30.967731 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5df8f4b68b-4wtbz" event={"ID":"2b6d5797-cca7-4551-9caf-bf984e82a0c3","Type":"ContainerDied","Data":"74649c95ea5988796788d5150d367a66ce51ad4303b92cbba6836fc321660bbe"} Aug 04 10:31:30.967851 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:30.967756 2580 scope.go:117] "RemoveContainer" containerID="f0fc32af417ed4f3cdc1a8f9d12ee1f5eb8fc5eda54197f3a46e18b31895ad55" Aug 04 10:31:30.973866 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:30.973841 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b6d5797-cca7-4551-9caf-bf984e82a0c3-trusted-ca-bundle\") pod \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\" (UID: \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\") " Aug 04 10:31:30.973983 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:30.973883 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b6d5797-cca7-4551-9caf-bf984e82a0c3-oauth-serving-cert\") pod \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\" (UID: \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\") " Aug 04 10:31:30.973983 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:30.973926 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4hws\" (UniqueName: \"kubernetes.io/projected/2b6d5797-cca7-4551-9caf-bf984e82a0c3-kube-api-access-t4hws\") pod \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\" (UID: \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\") " Aug 04 10:31:30.973983 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:30.973946 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b6d5797-cca7-4551-9caf-bf984e82a0c3-console-serving-cert\") pod \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\" (UID: \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\") " Aug 04 10:31:30.973983 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:30.973960 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b6d5797-cca7-4551-9caf-bf984e82a0c3-console-config\") pod \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\" (UID: \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\") " Aug 04 10:31:30.974196 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:30.973984 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b6d5797-cca7-4551-9caf-bf984e82a0c3-service-ca\") pod \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\" (UID: \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\") " Aug 04 10:31:30.974196 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:30.974007 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b6d5797-cca7-4551-9caf-bf984e82a0c3-console-oauth-config\") pod \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\" (UID: \"2b6d5797-cca7-4551-9caf-bf984e82a0c3\") " Aug 04 10:31:30.974295 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:30.974229 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b6d5797-cca7-4551-9caf-bf984e82a0c3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2b6d5797-cca7-4551-9caf-bf984e82a0c3" (UID: "2b6d5797-cca7-4551-9caf-bf984e82a0c3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 10:31:30.974598 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:30.974542 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b6d5797-cca7-4551-9caf-bf984e82a0c3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2b6d5797-cca7-4551-9caf-bf984e82a0c3" (UID: "2b6d5797-cca7-4551-9caf-bf984e82a0c3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 10:31:30.974879 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:30.974608 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b6d5797-cca7-4551-9caf-bf984e82a0c3-console-config" (OuterVolumeSpecName: "console-config") pod "2b6d5797-cca7-4551-9caf-bf984e82a0c3" (UID: "2b6d5797-cca7-4551-9caf-bf984e82a0c3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 10:31:30.974879 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:30.974618 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b6d5797-cca7-4551-9caf-bf984e82a0c3-service-ca" (OuterVolumeSpecName: "service-ca") pod "2b6d5797-cca7-4551-9caf-bf984e82a0c3" (UID: "2b6d5797-cca7-4551-9caf-bf984e82a0c3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 10:31:30.976313 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:30.976290 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6d5797-cca7-4551-9caf-bf984e82a0c3-kube-api-access-t4hws" (OuterVolumeSpecName: "kube-api-access-t4hws") pod "2b6d5797-cca7-4551-9caf-bf984e82a0c3" (UID: "2b6d5797-cca7-4551-9caf-bf984e82a0c3"). InnerVolumeSpecName "kube-api-access-t4hws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:31:30.976417 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:30.976319 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b6d5797-cca7-4551-9caf-bf984e82a0c3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2b6d5797-cca7-4551-9caf-bf984e82a0c3" (UID: "2b6d5797-cca7-4551-9caf-bf984e82a0c3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 10:31:30.976417 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:30.976402 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b6d5797-cca7-4551-9caf-bf984e82a0c3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2b6d5797-cca7-4551-9caf-bf984e82a0c3" (UID: "2b6d5797-cca7-4551-9caf-bf984e82a0c3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 10:31:30.976666 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:30.976646 2580 scope.go:117] "RemoveContainer" containerID="f0fc32af417ed4f3cdc1a8f9d12ee1f5eb8fc5eda54197f3a46e18b31895ad55" Aug 04 10:31:30.976993 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:31:30.976973 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0fc32af417ed4f3cdc1a8f9d12ee1f5eb8fc5eda54197f3a46e18b31895ad55\": container with ID starting with f0fc32af417ed4f3cdc1a8f9d12ee1f5eb8fc5eda54197f3a46e18b31895ad55 not found: ID does not exist" containerID="f0fc32af417ed4f3cdc1a8f9d12ee1f5eb8fc5eda54197f3a46e18b31895ad55" Aug 04 10:31:30.977047 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:30.977002 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0fc32af417ed4f3cdc1a8f9d12ee1f5eb8fc5eda54197f3a46e18b31895ad55"} err="failed to get container status \"f0fc32af417ed4f3cdc1a8f9d12ee1f5eb8fc5eda54197f3a46e18b31895ad55\": rpc error: code = NotFound desc = could not find container \"f0fc32af417ed4f3cdc1a8f9d12ee1f5eb8fc5eda54197f3a46e18b31895ad55\": container with ID starting with f0fc32af417ed4f3cdc1a8f9d12ee1f5eb8fc5eda54197f3a46e18b31895ad55 not found: ID does not exist" Aug 04 10:31:31.074824 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:31.074789 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b6d5797-cca7-4551-9caf-bf984e82a0c3-oauth-serving-cert\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:31:31.074824 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:31.074814 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4hws\" (UniqueName: \"kubernetes.io/projected/2b6d5797-cca7-4551-9caf-bf984e82a0c3-kube-api-access-t4hws\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:31:31.074824 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:31.074825 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b6d5797-cca7-4551-9caf-bf984e82a0c3-console-serving-cert\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:31:31.074824 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:31.074835 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b6d5797-cca7-4551-9caf-bf984e82a0c3-console-config\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:31:31.075081 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:31.074843 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b6d5797-cca7-4551-9caf-bf984e82a0c3-service-ca\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:31:31.075081 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:31.074851 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b6d5797-cca7-4551-9caf-bf984e82a0c3-console-oauth-config\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:31:31.075081 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:31.074859 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b6d5797-cca7-4551-9caf-bf984e82a0c3-trusted-ca-bundle\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:31:31.284185 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:31.284093 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5df8f4b68b-4wtbz"] Aug 04 10:31:31.287860 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:31.287834 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5df8f4b68b-4wtbz"] Aug 04 10:31:33.097682 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:33.097652 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b6d5797-cca7-4551-9caf-bf984e82a0c3" path="/var/lib/kubelet/pods/2b6d5797-cca7-4551-9caf-bf984e82a0c3/volumes" Aug 04 10:31:48.984309 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:48.984269 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-s5ll7"] Aug 04 10:31:48.984794 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:48.984582 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b6d5797-cca7-4551-9caf-bf984e82a0c3" containerName="console" Aug 04 10:31:48.984794 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:48.984599 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b6d5797-cca7-4551-9caf-bf984e82a0c3" containerName="console" Aug 04 10:31:48.984794 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:48.984681 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b6d5797-cca7-4551-9caf-bf984e82a0c3" containerName="console" Aug 04 10:31:48.988856 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:48.988835 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-s5ll7" Aug 04 10:31:48.992072 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:48.992051 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-md7vf\"" Aug 04 10:31:48.993342 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:48.993326 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Aug 04 10:31:48.993411 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:48.993327 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Aug 04 10:31:49.000404 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:49.000383 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-s5ll7"] Aug 04 10:31:49.087718 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:49.087677 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf0439c8-e9d0-40dc-bc2f-b60f678ed740-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-s5ll7\" (UID: \"bf0439c8-e9d0-40dc-bc2f-b60f678ed740\") " pod="cert-manager/cert-manager-webhook-6486ffd997-s5ll7" Aug 04 10:31:49.087718 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:49.087728 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz9ld\" (UniqueName: \"kubernetes.io/projected/bf0439c8-e9d0-40dc-bc2f-b60f678ed740-kube-api-access-dz9ld\") pod \"cert-manager-webhook-6486ffd997-s5ll7\" (UID: \"bf0439c8-e9d0-40dc-bc2f-b60f678ed740\") " pod="cert-manager/cert-manager-webhook-6486ffd997-s5ll7" Aug 04 10:31:49.188680 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:49.188639 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf0439c8-e9d0-40dc-bc2f-b60f678ed740-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-s5ll7\" (UID: \"bf0439c8-e9d0-40dc-bc2f-b60f678ed740\") " pod="cert-manager/cert-manager-webhook-6486ffd997-s5ll7" Aug 04 10:31:49.188680 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:49.188686 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz9ld\" (UniqueName: \"kubernetes.io/projected/bf0439c8-e9d0-40dc-bc2f-b60f678ed740-kube-api-access-dz9ld\") pod \"cert-manager-webhook-6486ffd997-s5ll7\" (UID: \"bf0439c8-e9d0-40dc-bc2f-b60f678ed740\") " pod="cert-manager/cert-manager-webhook-6486ffd997-s5ll7" Aug 04 10:31:49.201172 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:49.201136 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf0439c8-e9d0-40dc-bc2f-b60f678ed740-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-s5ll7\" (UID: \"bf0439c8-e9d0-40dc-bc2f-b60f678ed740\") " pod="cert-manager/cert-manager-webhook-6486ffd997-s5ll7" Aug 04 10:31:49.201314 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:49.201297 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz9ld\" (UniqueName: \"kubernetes.io/projected/bf0439c8-e9d0-40dc-bc2f-b60f678ed740-kube-api-access-dz9ld\") pod \"cert-manager-webhook-6486ffd997-s5ll7\" (UID: \"bf0439c8-e9d0-40dc-bc2f-b60f678ed740\") " pod="cert-manager/cert-manager-webhook-6486ffd997-s5ll7" Aug 04 10:31:49.297614 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:49.297515 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-s5ll7" Aug 04 10:31:49.430964 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:49.430923 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-s5ll7"] Aug 04 10:31:49.434095 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:31:49.434062 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf0439c8_e9d0_40dc_bc2f_b60f678ed740.slice/crio-a41788d7cae127c4ef9552152bff614dac7dc3608dcb8e5d4d4778616edcb637 WatchSource:0}: Error finding container a41788d7cae127c4ef9552152bff614dac7dc3608dcb8e5d4d4778616edcb637: Status 404 returned error can't find the container with id a41788d7cae127c4ef9552152bff614dac7dc3608dcb8e5d4d4778616edcb637 Aug 04 10:31:50.030672 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:31:50.030633 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-s5ll7" event={"ID":"bf0439c8-e9d0-40dc-bc2f-b60f678ed740","Type":"ContainerStarted","Data":"a41788d7cae127c4ef9552152bff614dac7dc3608dcb8e5d4d4778616edcb637"} Aug 04 10:32:10.097440 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:10.097400 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-s5ll7" event={"ID":"bf0439c8-e9d0-40dc-bc2f-b60f678ed740","Type":"ContainerStarted","Data":"bdd78940d18dda53daf1cfb49dfaeae6a60cdfd3102623ae2d6a3a8f0543d3ec"} Aug 04 10:32:10.097873 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:10.097542 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-s5ll7" Aug 04 10:32:10.115950 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:10.115892 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-s5ll7" podStartSLOduration=1.5742995469999999 podStartE2EDuration="22.115790864s" totalImagesPullingTime="20.541491317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:31:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:32:10.11554883 +0000 UTC m=+269.536436241" watchObservedRunningTime="2026-08-04 10:32:10.115790864 +0000 UTC m=+269.536678276" Aug 04 10:32:16.103350 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:16.103320 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-s5ll7" Aug 04 10:32:20.994646 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:20.994611 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-598d67d848-6jswn"] Aug 04 10:32:20.997760 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:20.997742 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-6jswn" Aug 04 10:32:21.002434 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:21.002411 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Aug 04 10:32:21.002533 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:21.002482 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hd694\"" Aug 04 10:32:21.002818 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:21.002803 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Aug 04 10:32:21.003298 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:21.003278 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Aug 04 10:32:21.004090 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:21.003923 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Aug 04 10:32:21.024278 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:21.024242 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-598d67d848-6jswn"] Aug 04 10:32:21.112763 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:21.112724 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6765da31-9d18-4e54-bdc9-886576264de3-manifests\") pod \"opendatahub-operator-controller-manager-598d67d848-6jswn\" (UID: \"6765da31-9d18-4e54-bdc9-886576264de3\") " pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-6jswn" Aug 04 10:32:21.112763 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:21.112759 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6765da31-9d18-4e54-bdc9-886576264de3-tmp\") pod \"opendatahub-operator-controller-manager-598d67d848-6jswn\" (UID: \"6765da31-9d18-4e54-bdc9-886576264de3\") " pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-6jswn" Aug 04 10:32:21.112969 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:21.112793 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9nnf\" (UniqueName: \"kubernetes.io/projected/6765da31-9d18-4e54-bdc9-886576264de3-kube-api-access-b9nnf\") pod \"opendatahub-operator-controller-manager-598d67d848-6jswn\" (UID: \"6765da31-9d18-4e54-bdc9-886576264de3\") " pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-6jswn" Aug 04 10:32:21.112969 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:21.112886 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6765da31-9d18-4e54-bdc9-886576264de3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-598d67d848-6jswn\" (UID: \"6765da31-9d18-4e54-bdc9-886576264de3\") " pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-6jswn" Aug 04 10:32:21.112969 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:21.112914 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6765da31-9d18-4e54-bdc9-886576264de3-webhook-cert\") pod \"opendatahub-operator-controller-manager-598d67d848-6jswn\" (UID: \"6765da31-9d18-4e54-bdc9-886576264de3\") " pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-6jswn" Aug 04 10:32:21.213811 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:21.213774 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6765da31-9d18-4e54-bdc9-886576264de3-manifests\") pod \"opendatahub-operator-controller-manager-598d67d848-6jswn\" (UID: \"6765da31-9d18-4e54-bdc9-886576264de3\") " pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-6jswn" Aug 04 10:32:21.213811 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:21.213809 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6765da31-9d18-4e54-bdc9-886576264de3-tmp\") pod \"opendatahub-operator-controller-manager-598d67d848-6jswn\" (UID: \"6765da31-9d18-4e54-bdc9-886576264de3\") " pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-6jswn" Aug 04 10:32:21.214034 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:21.213833 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9nnf\" (UniqueName: \"kubernetes.io/projected/6765da31-9d18-4e54-bdc9-886576264de3-kube-api-access-b9nnf\") pod \"opendatahub-operator-controller-manager-598d67d848-6jswn\" (UID: \"6765da31-9d18-4e54-bdc9-886576264de3\") " pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-6jswn" Aug 04 10:32:21.214034 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:21.213881 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6765da31-9d18-4e54-bdc9-886576264de3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-598d67d848-6jswn\" (UID: \"6765da31-9d18-4e54-bdc9-886576264de3\") " pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-6jswn" Aug 04 10:32:21.214034 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:21.213905 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6765da31-9d18-4e54-bdc9-886576264de3-webhook-cert\") pod \"opendatahub-operator-controller-manager-598d67d848-6jswn\" (UID: \"6765da31-9d18-4e54-bdc9-886576264de3\") " pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-6jswn" Aug 04 10:32:21.214181 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:21.214151 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6765da31-9d18-4e54-bdc9-886576264de3-manifests\") pod \"opendatahub-operator-controller-manager-598d67d848-6jswn\" (UID: \"6765da31-9d18-4e54-bdc9-886576264de3\") " pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-6jswn" Aug 04 10:32:21.216363 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:21.216333 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6765da31-9d18-4e54-bdc9-886576264de3-tmp\") pod \"opendatahub-operator-controller-manager-598d67d848-6jswn\" (UID: \"6765da31-9d18-4e54-bdc9-886576264de3\") " pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-6jswn" Aug 04 10:32:21.216478 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:21.216460 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6765da31-9d18-4e54-bdc9-886576264de3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-598d67d848-6jswn\" (UID: \"6765da31-9d18-4e54-bdc9-886576264de3\") " pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-6jswn" Aug 04 10:32:21.216521 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:21.216478 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6765da31-9d18-4e54-bdc9-886576264de3-webhook-cert\") pod \"opendatahub-operator-controller-manager-598d67d848-6jswn\" (UID: \"6765da31-9d18-4e54-bdc9-886576264de3\") " pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-6jswn" Aug 04 10:32:21.225419 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:21.225384 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9nnf\" (UniqueName: \"kubernetes.io/projected/6765da31-9d18-4e54-bdc9-886576264de3-kube-api-access-b9nnf\") pod \"opendatahub-operator-controller-manager-598d67d848-6jswn\" (UID: \"6765da31-9d18-4e54-bdc9-886576264de3\") " pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-6jswn" Aug 04 10:32:21.308165 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:21.308067 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-6jswn" Aug 04 10:32:21.446304 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:21.446267 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-598d67d848-6jswn"] Aug 04 10:32:21.449878 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:32:21.449850 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6765da31_9d18_4e54_bdc9_886576264de3.slice/crio-5a1ff4561deea15314211457e984dc6cbf8936b2cdb83d4827ab6826479b058c WatchSource:0}: Error finding container 5a1ff4561deea15314211457e984dc6cbf8936b2cdb83d4827ab6826479b058c: Status 404 returned error can't find the container with id 5a1ff4561deea15314211457e984dc6cbf8936b2cdb83d4827ab6826479b058c Aug 04 10:32:22.140887 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:22.140844 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-6jswn" event={"ID":"6765da31-9d18-4e54-bdc9-886576264de3","Type":"ContainerStarted","Data":"5a1ff4561deea15314211457e984dc6cbf8936b2cdb83d4827ab6826479b058c"} Aug 04 10:32:25.155976 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:25.155938 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-6jswn" event={"ID":"6765da31-9d18-4e54-bdc9-886576264de3","Type":"ContainerStarted","Data":"1892b544a0c8a6ee4b0c685300160b14cdf936c9cba27f31d2b8817da1a456b7"} Aug 04 10:32:26.160530 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:26.160492 2580 generic.go:356] "Generic (PLEG): container finished" podID="6765da31-9d18-4e54-bdc9-886576264de3" containerID="1892b544a0c8a6ee4b0c685300160b14cdf936c9cba27f31d2b8817da1a456b7" exitCode=0 Aug 04 10:32:26.160896 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:26.160587 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-6jswn" event={"ID":"6765da31-9d18-4e54-bdc9-886576264de3","Type":"ContainerDied","Data":"1892b544a0c8a6ee4b0c685300160b14cdf936c9cba27f31d2b8817da1a456b7"} Aug 04 10:32:27.165097 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:27.165063 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-6jswn" event={"ID":"6765da31-9d18-4e54-bdc9-886576264de3","Type":"ContainerStarted","Data":"d4422e4e3c40780686d1d19a98ff7ea2402a856cc289c7aa1164748eb4400eb3"} Aug 04 10:32:27.165489 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:27.165165 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-6jswn" Aug 04 10:32:27.196071 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:27.196024 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-6jswn" podStartSLOduration=2.919988101 podStartE2EDuration="7.196006967s" totalImagesPullingTime="3.238054295s" totalInitContainerRuntime="1.037964571s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:32:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:32:27.194237162 +0000 UTC m=+286.615124574" watchObservedRunningTime="2026-08-04 10:32:27.196006967 +0000 UTC m=+286.616894378" Aug 04 10:32:38.170710 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:38.170682 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-6jswn" Aug 04 10:32:40.246437 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.246398 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5bd95bfc5-zdsld"] Aug 04 10:32:40.338846 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.338801 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5bd95bfc5-zdsld"] Aug 04 10:32:40.339007 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.338901 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5bd95bfc5-zdsld" Aug 04 10:32:40.341912 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.341885 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Aug 04 10:32:40.342216 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.342200 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Aug 04 10:32:40.343403 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.343380 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Aug 04 10:32:40.343403 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.343397 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vxqnk\"" Aug 04 10:32:40.343553 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.343461 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Aug 04 10:32:40.377514 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.377482 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-hjxw2"] Aug 04 10:32:40.395009 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.394972 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-hjxw2"] Aug 04 10:32:40.395159 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.395129 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-hjxw2" Aug 04 10:32:40.397912 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.397876 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Aug 04 10:32:40.398201 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.398182 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-ht56z\"" Aug 04 10:32:40.456816 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.456788 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptn7z\" (UniqueName: \"kubernetes.io/projected/b112fb0c-d3f1-43c6-bef8-607a04229e08-kube-api-access-ptn7z\") pod \"kube-auth-proxy-5bd95bfc5-zdsld\" (UID: \"b112fb0c-d3f1-43c6-bef8-607a04229e08\") " pod="openshift-ingress/kube-auth-proxy-5bd95bfc5-zdsld" Aug 04 10:32:40.456980 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.456836 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b112fb0c-d3f1-43c6-bef8-607a04229e08-tmp\") pod \"kube-auth-proxy-5bd95bfc5-zdsld\" (UID: \"b112fb0c-d3f1-43c6-bef8-607a04229e08\") " pod="openshift-ingress/kube-auth-proxy-5bd95bfc5-zdsld" Aug 04 10:32:40.456980 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.456855 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b36de69a-8276-4697-a3b3-194f302385d7-cert\") pod \"odh-model-controller-7998c67dc4-hjxw2\" (UID: \"b36de69a-8276-4697-a3b3-194f302385d7\") " pod="opendatahub/odh-model-controller-7998c67dc4-hjxw2" Aug 04 10:32:40.456980 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.456874 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b112fb0c-d3f1-43c6-bef8-607a04229e08-tls-certs\") pod \"kube-auth-proxy-5bd95bfc5-zdsld\" (UID: \"b112fb0c-d3f1-43c6-bef8-607a04229e08\") " pod="openshift-ingress/kube-auth-proxy-5bd95bfc5-zdsld" Aug 04 10:32:40.456980 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.456914 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7rln\" (UniqueName: \"kubernetes.io/projected/b36de69a-8276-4697-a3b3-194f302385d7-kube-api-access-f7rln\") pod \"odh-model-controller-7998c67dc4-hjxw2\" (UID: \"b36de69a-8276-4697-a3b3-194f302385d7\") " pod="opendatahub/odh-model-controller-7998c67dc4-hjxw2" Aug 04 10:32:40.557781 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.557681 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b112fb0c-d3f1-43c6-bef8-607a04229e08-tmp\") pod \"kube-auth-proxy-5bd95bfc5-zdsld\" (UID: \"b112fb0c-d3f1-43c6-bef8-607a04229e08\") " pod="openshift-ingress/kube-auth-proxy-5bd95bfc5-zdsld" Aug 04 10:32:40.557781 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.557714 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b36de69a-8276-4697-a3b3-194f302385d7-cert\") pod \"odh-model-controller-7998c67dc4-hjxw2\" (UID: \"b36de69a-8276-4697-a3b3-194f302385d7\") " pod="opendatahub/odh-model-controller-7998c67dc4-hjxw2" Aug 04 10:32:40.557781 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.557734 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b112fb0c-d3f1-43c6-bef8-607a04229e08-tls-certs\") pod \"kube-auth-proxy-5bd95bfc5-zdsld\" (UID: \"b112fb0c-d3f1-43c6-bef8-607a04229e08\") " pod="openshift-ingress/kube-auth-proxy-5bd95bfc5-zdsld" Aug 04 10:32:40.557781 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.557762 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7rln\" (UniqueName: \"kubernetes.io/projected/b36de69a-8276-4697-a3b3-194f302385d7-kube-api-access-f7rln\") pod \"odh-model-controller-7998c67dc4-hjxw2\" (UID: \"b36de69a-8276-4697-a3b3-194f302385d7\") " pod="opendatahub/odh-model-controller-7998c67dc4-hjxw2" Aug 04 10:32:40.558052 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.557872 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptn7z\" (UniqueName: \"kubernetes.io/projected/b112fb0c-d3f1-43c6-bef8-607a04229e08-kube-api-access-ptn7z\") pod \"kube-auth-proxy-5bd95bfc5-zdsld\" (UID: \"b112fb0c-d3f1-43c6-bef8-607a04229e08\") " pod="openshift-ingress/kube-auth-proxy-5bd95bfc5-zdsld" Aug 04 10:32:40.560400 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.560373 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b112fb0c-d3f1-43c6-bef8-607a04229e08-tmp\") pod \"kube-auth-proxy-5bd95bfc5-zdsld\" (UID: \"b112fb0c-d3f1-43c6-bef8-607a04229e08\") " pod="openshift-ingress/kube-auth-proxy-5bd95bfc5-zdsld" Aug 04 10:32:40.560542 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.560463 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b112fb0c-d3f1-43c6-bef8-607a04229e08-tls-certs\") pod \"kube-auth-proxy-5bd95bfc5-zdsld\" (UID: \"b112fb0c-d3f1-43c6-bef8-607a04229e08\") " pod="openshift-ingress/kube-auth-proxy-5bd95bfc5-zdsld" Aug 04 10:32:40.560633 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.560582 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b36de69a-8276-4697-a3b3-194f302385d7-cert\") pod \"odh-model-controller-7998c67dc4-hjxw2\" (UID: \"b36de69a-8276-4697-a3b3-194f302385d7\") " pod="opendatahub/odh-model-controller-7998c67dc4-hjxw2" Aug 04 10:32:40.570930 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.570904 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptn7z\" (UniqueName: \"kubernetes.io/projected/b112fb0c-d3f1-43c6-bef8-607a04229e08-kube-api-access-ptn7z\") pod \"kube-auth-proxy-5bd95bfc5-zdsld\" (UID: \"b112fb0c-d3f1-43c6-bef8-607a04229e08\") " pod="openshift-ingress/kube-auth-proxy-5bd95bfc5-zdsld" Aug 04 10:32:40.571059 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.570912 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7rln\" (UniqueName: \"kubernetes.io/projected/b36de69a-8276-4697-a3b3-194f302385d7-kube-api-access-f7rln\") pod \"odh-model-controller-7998c67dc4-hjxw2\" (UID: \"b36de69a-8276-4697-a3b3-194f302385d7\") " pod="opendatahub/odh-model-controller-7998c67dc4-hjxw2" Aug 04 10:32:40.648957 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.648917 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5bd95bfc5-zdsld" Aug 04 10:32:40.707982 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.707947 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-hjxw2" Aug 04 10:32:40.809184 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.809147 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5bd95bfc5-zdsld"] Aug 04 10:32:40.811666 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:32:40.811636 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb112fb0c_d3f1_43c6_bef8_607a04229e08.slice/crio-8fd42fdd57d71ca6345e5a3b60ad55cb63f41a9d40e9cfaf6832a277cc6a0147 WatchSource:0}: Error finding container 8fd42fdd57d71ca6345e5a3b60ad55cb63f41a9d40e9cfaf6832a277cc6a0147: Status 404 returned error can't find the container with id 8fd42fdd57d71ca6345e5a3b60ad55cb63f41a9d40e9cfaf6832a277cc6a0147 Aug 04 10:32:40.882868 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.882838 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-hjxw2"] Aug 04 10:32:40.903242 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:32:40.903215 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb36de69a_8276_4697_a3b3_194f302385d7.slice/crio-c762faf6c5034adef200389e49993e178e54fdf61cb65d2b917d69481980b4e8 WatchSource:0}: Error finding container c762faf6c5034adef200389e49993e178e54fdf61cb65d2b917d69481980b4e8: Status 404 returned error can't find the container with id c762faf6c5034adef200389e49993e178e54fdf61cb65d2b917d69481980b4e8 Aug 04 10:32:40.946536 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.946503 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vq67x"] Aug 04 10:32:40.959939 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.959903 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tb4zk_c5922147-cb42-4f62-9880-8deaa51a4b0f/ovn-acl-logging/0.log" Aug 04 10:32:40.961070 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.961046 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tb4zk_c5922147-cb42-4f62-9880-8deaa51a4b0f/ovn-acl-logging/0.log" Aug 04 10:32:40.969239 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.969213 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vq67x" Aug 04 10:32:40.969618 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.969598 2580 kubelet.go:1648] "Image garbage collection succeeded" Aug 04 10:32:40.973084 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.973062 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Aug 04 10:32:40.974832 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:40.974807 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vq67x"] Aug 04 10:32:41.062300 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.062213 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9ftk\" (UniqueName: \"kubernetes.io/projected/f44b71f1-14e3-420e-b826-0705280ac08f-kube-api-access-g9ftk\") pod \"model-serving-api-54d89bd5fc-vq67x\" (UID: \"f44b71f1-14e3-420e-b826-0705280ac08f\") " pod="opendatahub/model-serving-api-54d89bd5fc-vq67x" Aug 04 10:32:41.062300 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.062274 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f44b71f1-14e3-420e-b826-0705280ac08f-tls-certs\") pod \"model-serving-api-54d89bd5fc-vq67x\" (UID: \"f44b71f1-14e3-420e-b826-0705280ac08f\") " pod="opendatahub/model-serving-api-54d89bd5fc-vq67x" Aug 04 10:32:41.163638 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.163604 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9ftk\" (UniqueName: \"kubernetes.io/projected/f44b71f1-14e3-420e-b826-0705280ac08f-kube-api-access-g9ftk\") pod \"model-serving-api-54d89bd5fc-vq67x\" (UID: \"f44b71f1-14e3-420e-b826-0705280ac08f\") " pod="opendatahub/model-serving-api-54d89bd5fc-vq67x" Aug 04 10:32:41.164793 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.163650 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f44b71f1-14e3-420e-b826-0705280ac08f-tls-certs\") pod \"model-serving-api-54d89bd5fc-vq67x\" (UID: \"f44b71f1-14e3-420e-b826-0705280ac08f\") " pod="opendatahub/model-serving-api-54d89bd5fc-vq67x" Aug 04 10:32:41.164793 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:32:41.163754 2580 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Aug 04 10:32:41.164793 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:32:41.163810 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f44b71f1-14e3-420e-b826-0705280ac08f-tls-certs podName:f44b71f1-14e3-420e-b826-0705280ac08f nodeName:}" failed. No retries permitted until 2026-08-04 10:32:41.663792798 +0000 UTC m=+301.084680187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f44b71f1-14e3-420e-b826-0705280ac08f-tls-certs") pod "model-serving-api-54d89bd5fc-vq67x" (UID: "f44b71f1-14e3-420e-b826-0705280ac08f") : secret "model-serving-api-tls" not found Aug 04 10:32:41.175246 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.175220 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9ftk\" (UniqueName: \"kubernetes.io/projected/f44b71f1-14e3-420e-b826-0705280ac08f-kube-api-access-g9ftk\") pod \"model-serving-api-54d89bd5fc-vq67x\" (UID: \"f44b71f1-14e3-420e-b826-0705280ac08f\") " pod="opendatahub/model-serving-api-54d89bd5fc-vq67x" Aug 04 10:32:41.212629 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.212593 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5bd95bfc5-zdsld" event={"ID":"b112fb0c-d3f1-43c6-bef8-607a04229e08","Type":"ContainerStarted","Data":"8fd42fdd57d71ca6345e5a3b60ad55cb63f41a9d40e9cfaf6832a277cc6a0147"} Aug 04 10:32:41.213610 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.213588 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-hjxw2" event={"ID":"b36de69a-8276-4697-a3b3-194f302385d7","Type":"ContainerStarted","Data":"c762faf6c5034adef200389e49993e178e54fdf61cb65d2b917d69481980b4e8"} Aug 04 10:32:41.633355 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.633316 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5"] Aug 04 10:32:41.635784 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.635757 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" Aug 04 10:32:41.638888 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.638670 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Aug 04 10:32:41.639132 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.639102 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Aug 04 10:32:41.639299 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.639173 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dq2q9\"" Aug 04 10:32:41.639379 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.639301 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Aug 04 10:32:41.639599 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.639581 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Aug 04 10:32:41.652477 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.651947 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5"] Aug 04 10:32:41.668593 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.668537 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f44b71f1-14e3-420e-b826-0705280ac08f-tls-certs\") pod \"model-serving-api-54d89bd5fc-vq67x\" (UID: \"f44b71f1-14e3-420e-b826-0705280ac08f\") " pod="opendatahub/model-serving-api-54d89bd5fc-vq67x" Aug 04 10:32:41.674633 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.674606 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f44b71f1-14e3-420e-b826-0705280ac08f-tls-certs\") pod \"model-serving-api-54d89bd5fc-vq67x\" (UID: \"f44b71f1-14e3-420e-b826-0705280ac08f\") " pod="opendatahub/model-serving-api-54d89bd5fc-vq67x" Aug 04 10:32:41.769462 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.769420 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-9kqk5\" (UID: \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" Aug 04 10:32:41.769674 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.769487 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-9kqk5\" (UID: \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" Aug 04 10:32:41.769674 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.769519 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-9kqk5\" (UID: \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" Aug 04 10:32:41.769674 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.769552 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-9kqk5\" (UID: \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" Aug 04 10:32:41.769674 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.769605 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-9kqk5\" (UID: \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" Aug 04 10:32:41.769674 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.769657 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-9kqk5\" (UID: \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" Aug 04 10:32:41.769946 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.769692 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p49t\" (UniqueName: \"kubernetes.io/projected/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-kube-api-access-9p49t\") pod \"istiod-openshift-gateway-7b7995d5cd-9kqk5\" (UID: \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" Aug 04 10:32:41.871131 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.871096 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-9kqk5\" (UID: \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" Aug 04 10:32:41.871321 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.871151 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p49t\" (UniqueName: \"kubernetes.io/projected/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-kube-api-access-9p49t\") pod \"istiod-openshift-gateway-7b7995d5cd-9kqk5\" (UID: \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" Aug 04 10:32:41.871321 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.871208 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-9kqk5\" (UID: \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" Aug 04 10:32:41.871321 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.871241 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-9kqk5\" (UID: \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" Aug 04 10:32:41.871321 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.871272 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-9kqk5\" (UID: \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" Aug 04 10:32:41.871321 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.871302 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-9kqk5\" (UID: \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" Aug 04 10:32:41.871656 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.871336 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-9kqk5\" (UID: \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" Aug 04 10:32:41.872242 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.872211 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-9kqk5\" (UID: \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" Aug 04 10:32:41.875102 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.875040 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-9kqk5\" (UID: \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" Aug 04 10:32:41.875921 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.875897 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-9kqk5\" (UID: \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" Aug 04 10:32:41.876611 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.876589 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-9kqk5\" (UID: \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" Aug 04 10:32:41.877469 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.877427 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-9kqk5\" (UID: \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" Aug 04 10:32:41.881816 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.881628 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vq67x" Aug 04 10:32:41.887201 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.887176 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p49t\" (UniqueName: \"kubernetes.io/projected/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-kube-api-access-9p49t\") pod \"istiod-openshift-gateway-7b7995d5cd-9kqk5\" (UID: \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" Aug 04 10:32:41.888910 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.888860 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-9kqk5\" (UID: \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" Aug 04 10:32:41.950793 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:41.950372 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" Aug 04 10:32:42.067262 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:42.067158 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vq67x"] Aug 04 10:32:42.076459 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:42.076440 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 10:32:42.165528 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:42.165436 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5"] Aug 04 10:32:42.171468 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:32:42.171433 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad42f7f0_9ac9_4ab3_8732_1c7a619eb98a.slice/crio-45952f3774991380fcc2961c620e2e2ec0ff35d56837d31ab9d5405315059e2a WatchSource:0}: Error finding container 45952f3774991380fcc2961c620e2e2ec0ff35d56837d31ab9d5405315059e2a: Status 404 returned error can't find the container with id 45952f3774991380fcc2961c620e2e2ec0ff35d56837d31ab9d5405315059e2a Aug 04 10:32:42.222057 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:42.221958 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" event={"ID":"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a","Type":"ContainerStarted","Data":"45952f3774991380fcc2961c620e2e2ec0ff35d56837d31ab9d5405315059e2a"} Aug 04 10:32:42.223519 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:42.223449 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vq67x" event={"ID":"f44b71f1-14e3-420e-b826-0705280ac08f","Type":"ContainerStarted","Data":"59e9314bfcf00f65e3b1b88f0c0f1fb53ccdf767b14de35f5564cbd148623ecd"} Aug 04 10:32:45.677905 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:45.677870 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Aug 04 10:32:46.241991 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:46.241948 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5bd95bfc5-zdsld" event={"ID":"b112fb0c-d3f1-43c6-bef8-607a04229e08","Type":"ContainerStarted","Data":"0c02b44e632585f0a9f3606df78e4398af0321cff67822f57dc5acd81b52672b"} Aug 04 10:32:46.243498 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:46.243476 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-hjxw2" event={"ID":"b36de69a-8276-4697-a3b3-194f302385d7","Type":"ContainerStarted","Data":"70e1e731295ea262ad2d4f0f4831917dc22744eb3164e4346e00851b0ea93ee3"} Aug 04 10:32:46.243622 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:46.243540 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-hjxw2" Aug 04 10:32:46.244834 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:46.244815 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vq67x" event={"ID":"f44b71f1-14e3-420e-b826-0705280ac08f","Type":"ContainerStarted","Data":"a0161ab034b1a59231020471bc341e95f64620d82090a37363b74a37751dce61"} Aug 04 10:32:46.244971 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:46.244954 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-vq67x" Aug 04 10:32:46.261149 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:46.261103 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5bd95bfc5-zdsld" podStartSLOduration=1.40045267 podStartE2EDuration="6.26108776s" totalImagesPullingTime="4.86063509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:32:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:32:46.261003902 +0000 UTC m=+305.681891313" watchObservedRunningTime="2026-08-04 10:32:46.26108776 +0000 UTC m=+305.681975172" Aug 04 10:32:46.281062 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:46.281003 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-hjxw2" podStartSLOduration=1.491810215 podStartE2EDuration="6.280982822s" totalImagesPullingTime="4.789172607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:32:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:32:46.279791155 +0000 UTC m=+305.700678568" watchObservedRunningTime="2026-08-04 10:32:46.280982822 +0000 UTC m=+305.701870235" Aug 04 10:32:46.297378 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:46.297331 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-vq67x" podStartSLOduration=2.686853948 podStartE2EDuration="6.297316553s" totalImagesPullingTime="3.610462605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:32:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:32:46.296218342 +0000 UTC m=+305.717105752" watchObservedRunningTime="2026-08-04 10:32:46.297316553 +0000 UTC m=+305.718203964" Aug 04 10:32:50.721292 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:50.721253 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-wnv4x"] Aug 04 10:32:50.724598 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:50.724555 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wnv4x" Aug 04 10:32:50.727839 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:50.727813 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-kl2rb"] Aug 04 10:32:50.729117 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:50.729095 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-g547f\"" Aug 04 10:32:50.729422 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:50.729394 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Aug 04 10:32:50.730976 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:50.730956 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kl2rb" Aug 04 10:32:50.734488 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:50.734468 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Aug 04 10:32:50.734608 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:50.734501 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-w9j2w\"" Aug 04 10:32:50.739312 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:50.739290 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-wnv4x"] Aug 04 10:32:50.747167 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:50.747145 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-kl2rb"] Aug 04 10:32:50.842307 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:50.842270 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b6a041d-9d50-4a1e-bbfa-527e5859d8cd-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-kl2rb\" (UID: \"7b6a041d-9d50-4a1e-bbfa-527e5859d8cd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kl2rb" Aug 04 10:32:50.842307 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:50.842311 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px96l\" (UniqueName: \"kubernetes.io/projected/a4a17547-6158-4123-bc5f-39d76f32974a-kube-api-access-px96l\") pod \"kserve-controller-manager-d6f4b65b8-wnv4x\" (UID: \"a4a17547-6158-4123-bc5f-39d76f32974a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wnv4x" Aug 04 10:32:50.842525 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:50.842380 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a4a17547-6158-4123-bc5f-39d76f32974a-cert\") pod \"kserve-controller-manager-d6f4b65b8-wnv4x\" (UID: \"a4a17547-6158-4123-bc5f-39d76f32974a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wnv4x" Aug 04 10:32:50.842525 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:50.842438 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8m88\" (UniqueName: \"kubernetes.io/projected/7b6a041d-9d50-4a1e-bbfa-527e5859d8cd-kube-api-access-n8m88\") pod \"llmisvc-controller-manager-777c6bd7d5-kl2rb\" (UID: \"7b6a041d-9d50-4a1e-bbfa-527e5859d8cd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kl2rb" Aug 04 10:32:50.943146 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:50.943107 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b6a041d-9d50-4a1e-bbfa-527e5859d8cd-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-kl2rb\" (UID: \"7b6a041d-9d50-4a1e-bbfa-527e5859d8cd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kl2rb" Aug 04 10:32:50.943146 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:50.943148 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px96l\" (UniqueName: \"kubernetes.io/projected/a4a17547-6158-4123-bc5f-39d76f32974a-kube-api-access-px96l\") pod \"kserve-controller-manager-d6f4b65b8-wnv4x\" (UID: \"a4a17547-6158-4123-bc5f-39d76f32974a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wnv4x" Aug 04 10:32:50.943325 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:50.943172 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a4a17547-6158-4123-bc5f-39d76f32974a-cert\") pod \"kserve-controller-manager-d6f4b65b8-wnv4x\" (UID: \"a4a17547-6158-4123-bc5f-39d76f32974a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wnv4x" Aug 04 10:32:50.943325 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:50.943203 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8m88\" (UniqueName: \"kubernetes.io/projected/7b6a041d-9d50-4a1e-bbfa-527e5859d8cd-kube-api-access-n8m88\") pod \"llmisvc-controller-manager-777c6bd7d5-kl2rb\" (UID: \"7b6a041d-9d50-4a1e-bbfa-527e5859d8cd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kl2rb" Aug 04 10:32:50.943325 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:32:50.943277 2580 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Aug 04 10:32:50.943422 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:32:50.943352 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4a17547-6158-4123-bc5f-39d76f32974a-cert podName:a4a17547-6158-4123-bc5f-39d76f32974a nodeName:}" failed. No retries permitted until 2026-08-04 10:32:51.443331752 +0000 UTC m=+310.864219143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a4a17547-6158-4123-bc5f-39d76f32974a-cert") pod "kserve-controller-manager-d6f4b65b8-wnv4x" (UID: "a4a17547-6158-4123-bc5f-39d76f32974a") : secret "kserve-webhook-server-cert" not found Aug 04 10:32:50.945698 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:50.945670 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b6a041d-9d50-4a1e-bbfa-527e5859d8cd-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-kl2rb\" (UID: \"7b6a041d-9d50-4a1e-bbfa-527e5859d8cd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kl2rb" Aug 04 10:32:50.957461 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:50.957435 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8m88\" (UniqueName: \"kubernetes.io/projected/7b6a041d-9d50-4a1e-bbfa-527e5859d8cd-kube-api-access-n8m88\") pod \"llmisvc-controller-manager-777c6bd7d5-kl2rb\" (UID: \"7b6a041d-9d50-4a1e-bbfa-527e5859d8cd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kl2rb" Aug 04 10:32:50.967115 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:50.967090 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px96l\" (UniqueName: \"kubernetes.io/projected/a4a17547-6158-4123-bc5f-39d76f32974a-kube-api-access-px96l\") pod \"kserve-controller-manager-d6f4b65b8-wnv4x\" (UID: \"a4a17547-6158-4123-bc5f-39d76f32974a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wnv4x" Aug 04 10:32:51.044297 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:51.044191 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kl2rb" Aug 04 10:32:51.186507 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:51.186482 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-kl2rb"] Aug 04 10:32:51.188993 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:32:51.188967 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7b6a041d_9d50_4a1e_bbfa_527e5859d8cd.slice/crio-96683104b4451420fc5fc0495e837208fa92a23291c8d79c7731eb59939fcf12 WatchSource:0}: Error finding container 96683104b4451420fc5fc0495e837208fa92a23291c8d79c7731eb59939fcf12: Status 404 returned error can't find the container with id 96683104b4451420fc5fc0495e837208fa92a23291c8d79c7731eb59939fcf12 Aug 04 10:32:51.263465 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:51.263431 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kl2rb" event={"ID":"7b6a041d-9d50-4a1e-bbfa-527e5859d8cd","Type":"ContainerStarted","Data":"96683104b4451420fc5fc0495e837208fa92a23291c8d79c7731eb59939fcf12"} Aug 04 10:32:51.448419 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:51.448370 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a4a17547-6158-4123-bc5f-39d76f32974a-cert\") pod \"kserve-controller-manager-d6f4b65b8-wnv4x\" (UID: \"a4a17547-6158-4123-bc5f-39d76f32974a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wnv4x" Aug 04 10:32:51.448678 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:32:51.448546 2580 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Aug 04 10:32:51.448678 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:32:51.448650 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4a17547-6158-4123-bc5f-39d76f32974a-cert podName:a4a17547-6158-4123-bc5f-39d76f32974a nodeName:}" failed. No retries permitted until 2026-08-04 10:32:52.448631268 +0000 UTC m=+311.869518670 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a4a17547-6158-4123-bc5f-39d76f32974a-cert") pod "kserve-controller-manager-d6f4b65b8-wnv4x" (UID: "a4a17547-6158-4123-bc5f-39d76f32974a") : secret "kserve-webhook-server-cert" not found Aug 04 10:32:52.457764 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.457723 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a4a17547-6158-4123-bc5f-39d76f32974a-cert\") pod \"kserve-controller-manager-d6f4b65b8-wnv4x\" (UID: \"a4a17547-6158-4123-bc5f-39d76f32974a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wnv4x" Aug 04 10:32:52.460897 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.460845 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a4a17547-6158-4123-bc5f-39d76f32974a-cert\") pod \"kserve-controller-manager-d6f4b65b8-wnv4x\" (UID: \"a4a17547-6158-4123-bc5f-39d76f32974a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wnv4x" Aug 04 10:32:52.536779 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.536734 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wnv4x" Aug 04 10:32:52.548540 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.548510 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv"] Aug 04 10:32:52.551846 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.551822 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:52.555832 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.555804 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-lf4l7\"" Aug 04 10:32:52.659649 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.659610 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsjlq\" (UniqueName: \"kubernetes.io/projected/7ee19c9c-fbf9-4f49-8520-0aedeee66aa6-kube-api-access-fsjlq\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv\" (UID: \"7ee19c9c-fbf9-4f49-8520-0aedeee66aa6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:52.659649 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.659651 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee19c9c-fbf9-4f49-8520-0aedeee66aa6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv\" (UID: \"7ee19c9c-fbf9-4f49-8520-0aedeee66aa6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:52.659821 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.659670 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee19c9c-fbf9-4f49-8520-0aedeee66aa6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv\" (UID: \"7ee19c9c-fbf9-4f49-8520-0aedeee66aa6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:52.659821 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.659749 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee19c9c-fbf9-4f49-8520-0aedeee66aa6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv\" (UID: \"7ee19c9c-fbf9-4f49-8520-0aedeee66aa6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:52.659821 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.659783 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee19c9c-fbf9-4f49-8520-0aedeee66aa6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv\" (UID: \"7ee19c9c-fbf9-4f49-8520-0aedeee66aa6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:52.659821 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.659811 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee19c9c-fbf9-4f49-8520-0aedeee66aa6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv\" (UID: \"7ee19c9c-fbf9-4f49-8520-0aedeee66aa6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:52.659973 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.659841 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee19c9c-fbf9-4f49-8520-0aedeee66aa6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv\" (UID: \"7ee19c9c-fbf9-4f49-8520-0aedeee66aa6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:52.659973 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.659894 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee19c9c-fbf9-4f49-8520-0aedeee66aa6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv\" (UID: \"7ee19c9c-fbf9-4f49-8520-0aedeee66aa6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:52.659973 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.659921 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee19c9c-fbf9-4f49-8520-0aedeee66aa6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv\" (UID: \"7ee19c9c-fbf9-4f49-8520-0aedeee66aa6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:52.695628 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.695598 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv"] Aug 04 10:32:52.761176 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.761096 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee19c9c-fbf9-4f49-8520-0aedeee66aa6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv\" (UID: \"7ee19c9c-fbf9-4f49-8520-0aedeee66aa6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:52.761176 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.761150 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee19c9c-fbf9-4f49-8520-0aedeee66aa6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv\" (UID: \"7ee19c9c-fbf9-4f49-8520-0aedeee66aa6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:52.761555 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.761319 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee19c9c-fbf9-4f49-8520-0aedeee66aa6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv\" (UID: \"7ee19c9c-fbf9-4f49-8520-0aedeee66aa6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:52.761555 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.761387 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsjlq\" (UniqueName: \"kubernetes.io/projected/7ee19c9c-fbf9-4f49-8520-0aedeee66aa6-kube-api-access-fsjlq\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv\" (UID: \"7ee19c9c-fbf9-4f49-8520-0aedeee66aa6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:52.761555 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.761497 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee19c9c-fbf9-4f49-8520-0aedeee66aa6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv\" (UID: \"7ee19c9c-fbf9-4f49-8520-0aedeee66aa6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:52.761555 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.761550 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee19c9c-fbf9-4f49-8520-0aedeee66aa6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv\" (UID: \"7ee19c9c-fbf9-4f49-8520-0aedeee66aa6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:52.761555 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.761595 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee19c9c-fbf9-4f49-8520-0aedeee66aa6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv\" (UID: \"7ee19c9c-fbf9-4f49-8520-0aedeee66aa6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:52.761973 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.761632 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee19c9c-fbf9-4f49-8520-0aedeee66aa6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv\" (UID: \"7ee19c9c-fbf9-4f49-8520-0aedeee66aa6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:52.761973 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.761662 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee19c9c-fbf9-4f49-8520-0aedeee66aa6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv\" (UID: \"7ee19c9c-fbf9-4f49-8520-0aedeee66aa6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:52.761973 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.761702 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee19c9c-fbf9-4f49-8520-0aedeee66aa6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv\" (UID: \"7ee19c9c-fbf9-4f49-8520-0aedeee66aa6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:52.761973 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.761717 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee19c9c-fbf9-4f49-8520-0aedeee66aa6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv\" (UID: \"7ee19c9c-fbf9-4f49-8520-0aedeee66aa6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:52.761973 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.761903 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee19c9c-fbf9-4f49-8520-0aedeee66aa6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv\" (UID: \"7ee19c9c-fbf9-4f49-8520-0aedeee66aa6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:52.762437 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.762057 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee19c9c-fbf9-4f49-8520-0aedeee66aa6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv\" (UID: \"7ee19c9c-fbf9-4f49-8520-0aedeee66aa6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:52.762547 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.762520 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee19c9c-fbf9-4f49-8520-0aedeee66aa6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv\" (UID: \"7ee19c9c-fbf9-4f49-8520-0aedeee66aa6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:52.764741 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.764714 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee19c9c-fbf9-4f49-8520-0aedeee66aa6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv\" (UID: \"7ee19c9c-fbf9-4f49-8520-0aedeee66aa6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:52.764955 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.764923 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee19c9c-fbf9-4f49-8520-0aedeee66aa6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv\" (UID: \"7ee19c9c-fbf9-4f49-8520-0aedeee66aa6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:52.771537 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:32:52.771498 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4a17547_6158_4123_bc5f_39d76f32974a.slice/crio-de7160127a67bc0b811077c7b18583a825c3b7196e9945219606290165928c06 WatchSource:0}: Error finding container de7160127a67bc0b811077c7b18583a825c3b7196e9945219606290165928c06: Status 404 returned error can't find the container with id de7160127a67bc0b811077c7b18583a825c3b7196e9945219606290165928c06 Aug 04 10:32:52.773312 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.773279 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-wnv4x"] Aug 04 10:32:52.784661 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.784638 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsjlq\" (UniqueName: \"kubernetes.io/projected/7ee19c9c-fbf9-4f49-8520-0aedeee66aa6-kube-api-access-fsjlq\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv\" (UID: \"7ee19c9c-fbf9-4f49-8520-0aedeee66aa6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:52.786718 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.786686 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee19c9c-fbf9-4f49-8520-0aedeee66aa6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv\" (UID: \"7ee19c9c-fbf9-4f49-8520-0aedeee66aa6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:52.862493 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.862455 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:52.891624 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:52.891586 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5"] Aug 04 10:32:53.014256 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:53.014228 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv"] Aug 04 10:32:53.016099 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:32:53.016069 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee19c9c_fbf9_4f49_8520_0aedeee66aa6.slice/crio-0c620df9aa77438f79345332d3394a0ca1279dd287eaa33036b573ee7f62719c WatchSource:0}: Error finding container 0c620df9aa77438f79345332d3394a0ca1279dd287eaa33036b573ee7f62719c: Status 404 returned error can't find the container with id 0c620df9aa77438f79345332d3394a0ca1279dd287eaa33036b573ee7f62719c Aug 04 10:32:53.274922 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:53.274833 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wnv4x" event={"ID":"a4a17547-6158-4123-bc5f-39d76f32974a","Type":"ContainerStarted","Data":"de7160127a67bc0b811077c7b18583a825c3b7196e9945219606290165928c06"} Aug 04 10:32:53.276071 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:53.276040 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" event={"ID":"7ee19c9c-fbf9-4f49-8520-0aedeee66aa6","Type":"ContainerStarted","Data":"0c620df9aa77438f79345332d3394a0ca1279dd287eaa33036b573ee7f62719c"} Aug 04 10:32:54.285581 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:54.285524 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kl2rb" event={"ID":"7b6a041d-9d50-4a1e-bbfa-527e5859d8cd","Type":"ContainerStarted","Data":"555596f5c31d2ab6725b77aef91aef14e767647ea835c0d1e27b618bfda81831"} Aug 04 10:32:54.286074 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:54.285771 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kl2rb" Aug 04 10:32:54.304642 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:54.304164 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kl2rb" podStartSLOduration=1.79210265 podStartE2EDuration="4.304141234s" totalImagesPullingTime="2.512038584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:32:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:32:54.303744796 +0000 UTC m=+313.724632212" watchObservedRunningTime="2026-08-04 10:32:54.304141234 +0000 UTC m=+313.725028647" Aug 04 10:32:56.574820 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:56.574773 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:32:56.575101 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:56.574870 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:32:56.575101 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:56.574913 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:32:57.252052 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:57.252013 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-hjxw2" Aug 04 10:32:57.254602 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:57.254577 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-vq67x" Aug 04 10:32:57.306406 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:57.306289 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" event={"ID":"7ee19c9c-fbf9-4f49-8520-0aedeee66aa6","Type":"ContainerStarted","Data":"a4f036507bb51ad0b9e3029b0b088cbb6e00835fb17fd7cb538b6171a4c89814"} Aug 04 10:32:57.308959 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:57.308928 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wnv4x" event={"ID":"a4a17547-6158-4123-bc5f-39d76f32974a","Type":"ContainerStarted","Data":"97457b910c1c31ef66e633ec10cc51d9829601427c4a10940f277bede5f9e6a8"} Aug 04 10:32:57.309150 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:57.309133 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wnv4x" Aug 04 10:32:57.331761 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:57.331696 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" podStartSLOduration=1.775334304 podStartE2EDuration="5.331673358s" totalImagesPullingTime="3.556339054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:32:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:32:57.330543311 +0000 UTC m=+316.751430724" watchObservedRunningTime="2026-08-04 10:32:57.331673358 +0000 UTC m=+316.752560771" Aug 04 10:32:57.352513 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:57.352461 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wnv4x" podStartSLOduration=3.55024575 podStartE2EDuration="7.352444339s" totalImagesPullingTime="3.802198589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:32:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:32:57.351285084 +0000 UTC m=+316.772172496" watchObservedRunningTime="2026-08-04 10:32:57.352444339 +0000 UTC m=+316.773331750" Aug 04 10:32:57.863366 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:57.863327 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:57.868951 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:57.868927 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:58.314360 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:58.314264 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:58.315208 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:58.315182 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv" Aug 04 10:32:58.446082 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:58.446046 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:32:58.446143 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:58.446121 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:32:59.320380 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:59.320321 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" event={"ID":"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a","Type":"ContainerStarted","Data":"57311a8ba5c677210bc5405765d05cfe9142f6c5263e8895f32312a83ee220c9"} Aug 04 10:32:59.320924 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:59.320370 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" podUID="ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a" containerName="discovery" containerID="cri-o://57311a8ba5c677210bc5405765d05cfe9142f6c5263e8895f32312a83ee220c9" gracePeriod=30 Aug 04 10:32:59.320924 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:59.320734 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" Aug 04 10:32:59.322770 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:59.322748 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" Aug 04 10:32:59.346770 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:59.346717 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" podStartSLOduration=2.075839635 podStartE2EDuration="18.346702256s" totalImagesPullingTime="16.270862621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:32:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:32:59.346132825 +0000 UTC m=+318.767020236" watchObservedRunningTime="2026-08-04 10:32:59.346702256 +0000 UTC m=+318.767589666" Aug 04 10:32:59.565359 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:59.565335 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" Aug 04 10:32:59.621909 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:59.621878 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-cacerts\") pod \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\" (UID: \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\") " Aug 04 10:32:59.621909 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:59.621916 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-istio-kubeconfig\") pod \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\" (UID: \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\") " Aug 04 10:32:59.622108 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:59.621941 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9p49t\" (UniqueName: \"kubernetes.io/projected/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-kube-api-access-9p49t\") pod \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\" (UID: \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\") " Aug 04 10:32:59.622108 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:59.621965 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-istio-token\") pod \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\" (UID: \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\") " Aug 04 10:32:59.622108 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:59.621997 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-local-certs\") pod \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\" (UID: \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\") " Aug 04 10:32:59.622108 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:59.622034 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-istio-csr-dns-cert\") pod \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\" (UID: \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\") " Aug 04 10:32:59.622108 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:59.622101 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-istio-csr-ca-configmap\") pod \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\" (UID: \"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a\") " Aug 04 10:32:59.622800 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:59.622770 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a" (UID: "ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 10:32:59.624374 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:59.624338 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-kube-api-access-9p49t" (OuterVolumeSpecName: "kube-api-access-9p49t") pod "ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a" (UID: "ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a"). InnerVolumeSpecName "kube-api-access-9p49t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:32:59.624485 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:59.624372 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-cacerts" (OuterVolumeSpecName: "cacerts") pod "ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a" (UID: "ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 10:32:59.624776 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:59.624744 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-local-certs" (OuterVolumeSpecName: "local-certs") pod "ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a" (UID: "ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:32:59.624776 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:59.624756 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-istio-token" (OuterVolumeSpecName: "istio-token") pod "ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a" (UID: "ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:32:59.624913 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:59.624810 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a" (UID: "ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 10:32:59.624913 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:59.624851 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a" (UID: "ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 10:32:59.722982 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:59.722946 2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-istio-csr-ca-configmap\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:32:59.722982 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:59.722974 2580 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-cacerts\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:32:59.722982 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:59.722983 2580 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-istio-kubeconfig\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:32:59.722982 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:59.722993 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p49t\" (UniqueName: \"kubernetes.io/projected/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-kube-api-access-9p49t\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:32:59.723232 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:59.723002 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-istio-token\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:32:59.723232 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:59.723010 2580 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-local-certs\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:32:59.723232 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:32:59.723017 2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a-istio-csr-dns-cert\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:33:00.325850 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:00.325814 2580 generic.go:356] "Generic (PLEG): container finished" podID="ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a" containerID="57311a8ba5c677210bc5405765d05cfe9142f6c5263e8895f32312a83ee220c9" exitCode=0 Aug 04 10:33:00.326263 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:00.325862 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" Aug 04 10:33:00.326263 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:00.325909 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" event={"ID":"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a","Type":"ContainerDied","Data":"57311a8ba5c677210bc5405765d05cfe9142f6c5263e8895f32312a83ee220c9"} Aug 04 10:33:00.326263 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:00.325959 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5" event={"ID":"ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a","Type":"ContainerDied","Data":"45952f3774991380fcc2961c620e2e2ec0ff35d56837d31ab9d5405315059e2a"} Aug 04 10:33:00.326263 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:00.325977 2580 scope.go:117] "RemoveContainer" containerID="57311a8ba5c677210bc5405765d05cfe9142f6c5263e8895f32312a83ee220c9" Aug 04 10:33:00.334294 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:00.334277 2580 scope.go:117] "RemoveContainer" containerID="57311a8ba5c677210bc5405765d05cfe9142f6c5263e8895f32312a83ee220c9" Aug 04 10:33:00.334612 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:33:00.334537 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57311a8ba5c677210bc5405765d05cfe9142f6c5263e8895f32312a83ee220c9\": container with ID starting with 57311a8ba5c677210bc5405765d05cfe9142f6c5263e8895f32312a83ee220c9 not found: ID does not exist" containerID="57311a8ba5c677210bc5405765d05cfe9142f6c5263e8895f32312a83ee220c9" Aug 04 10:33:00.334612 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:00.334582 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57311a8ba5c677210bc5405765d05cfe9142f6c5263e8895f32312a83ee220c9"} err="failed to get container status \"57311a8ba5c677210bc5405765d05cfe9142f6c5263e8895f32312a83ee220c9\": rpc error: code = NotFound desc = could not find container \"57311a8ba5c677210bc5405765d05cfe9142f6c5263e8895f32312a83ee220c9\": container with ID starting with 57311a8ba5c677210bc5405765d05cfe9142f6c5263e8895f32312a83ee220c9 not found: ID does not exist" Aug 04 10:33:00.348407 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:00.348384 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5"] Aug 04 10:33:00.353675 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:00.353654 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9kqk5"] Aug 04 10:33:01.098933 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:01.098901 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a" path="/var/lib/kubelet/pods/ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a/volumes" Aug 04 10:33:25.296015 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:25.295984 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kl2rb" Aug 04 10:33:28.319141 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:28.319113 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wnv4x" Aug 04 10:33:56.093983 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:56.093946 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-4npks"] Aug 04 10:33:56.094461 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:56.094211 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a" containerName="discovery" Aug 04 10:33:56.094461 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:56.094222 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a" containerName="discovery" Aug 04 10:33:56.094461 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:56.094288 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad42f7f0-9ac9-4ab3-8732-1c7a619eb98a" containerName="discovery" Aug 04 10:33:56.096678 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:56.096661 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-4npks" Aug 04 10:33:56.099775 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:56.099754 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 04 10:33:56.099904 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:56.099757 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Aug 04 10:33:56.099904 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:56.099837 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-p2w6c\"" Aug 04 10:33:56.099904 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:56.099870 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 04 10:33:56.109259 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:56.109234 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-4npks"] Aug 04 10:33:56.249138 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:56.249097 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwfrt\" (UniqueName: \"kubernetes.io/projected/2893cf39-ba4a-49bc-9660-c3123ae361ba-kube-api-access-wwfrt\") pod \"dns-operator-controller-manager-5985564776-4npks\" (UID: \"2893cf39-ba4a-49bc-9660-c3123ae361ba\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-4npks" Aug 04 10:33:56.350023 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:56.349943 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwfrt\" (UniqueName: \"kubernetes.io/projected/2893cf39-ba4a-49bc-9660-c3123ae361ba-kube-api-access-wwfrt\") pod \"dns-operator-controller-manager-5985564776-4npks\" (UID: \"2893cf39-ba4a-49bc-9660-c3123ae361ba\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-4npks" Aug 04 10:33:56.363652 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:56.363623 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwfrt\" (UniqueName: \"kubernetes.io/projected/2893cf39-ba4a-49bc-9660-c3123ae361ba-kube-api-access-wwfrt\") pod \"dns-operator-controller-manager-5985564776-4npks\" (UID: \"2893cf39-ba4a-49bc-9660-c3123ae361ba\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-4npks" Aug 04 10:33:56.406103 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:56.406073 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-4npks" Aug 04 10:33:56.544302 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:56.544149 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-4npks"] Aug 04 10:33:56.547173 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:33:56.547146 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2893cf39_ba4a_49bc_9660_c3123ae361ba.slice/crio-e460651c7359336d417db846b13ad4f5e646cbf599bef343914cb13e8ec78b32 WatchSource:0}: Error finding container e460651c7359336d417db846b13ad4f5e646cbf599bef343914cb13e8ec78b32: Status 404 returned error can't find the container with id e460651c7359336d417db846b13ad4f5e646cbf599bef343914cb13e8ec78b32 Aug 04 10:33:57.554461 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:57.554416 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-4npks" event={"ID":"2893cf39-ba4a-49bc-9660-c3123ae361ba","Type":"ContainerStarted","Data":"e460651c7359336d417db846b13ad4f5e646cbf599bef343914cb13e8ec78b32"} Aug 04 10:33:58.533932 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:58.533900 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-gwn2m"] Aug 04 10:33:58.536540 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:58.536520 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-gwn2m" Aug 04 10:33:58.539445 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:58.539421 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-4s2vj\"" Aug 04 10:33:58.550152 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:58.550122 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-gwn2m"] Aug 04 10:33:58.671609 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:58.671571 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bgr8\" (UniqueName: \"kubernetes.io/projected/b3f3b35a-3c47-4410-afd5-671953692c16-kube-api-access-5bgr8\") pod \"authorino-operator-68d88948d8-gwn2m\" (UID: \"b3f3b35a-3c47-4410-afd5-671953692c16\") " pod="kuadrant-system/authorino-operator-68d88948d8-gwn2m" Aug 04 10:33:58.772342 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:58.772306 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bgr8\" (UniqueName: \"kubernetes.io/projected/b3f3b35a-3c47-4410-afd5-671953692c16-kube-api-access-5bgr8\") pod \"authorino-operator-68d88948d8-gwn2m\" (UID: \"b3f3b35a-3c47-4410-afd5-671953692c16\") " pod="kuadrant-system/authorino-operator-68d88948d8-gwn2m" Aug 04 10:33:58.803045 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:58.802969 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bgr8\" (UniqueName: \"kubernetes.io/projected/b3f3b35a-3c47-4410-afd5-671953692c16-kube-api-access-5bgr8\") pod \"authorino-operator-68d88948d8-gwn2m\" (UID: \"b3f3b35a-3c47-4410-afd5-671953692c16\") " pod="kuadrant-system/authorino-operator-68d88948d8-gwn2m" Aug 04 10:33:58.847954 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:58.847920 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-gwn2m" Aug 04 10:33:59.001935 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:59.001911 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-gwn2m"] Aug 04 10:33:59.004061 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:33:59.004033 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3f3b35a_3c47_4410_afd5_671953692c16.slice/crio-aba046aa54bb08f26cf408fa8ed80a20bb00f087521aa1f46c558ff7f80e709a WatchSource:0}: Error finding container aba046aa54bb08f26cf408fa8ed80a20bb00f087521aa1f46c558ff7f80e709a: Status 404 returned error can't find the container with id aba046aa54bb08f26cf408fa8ed80a20bb00f087521aa1f46c558ff7f80e709a Aug 04 10:33:59.564130 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:59.564096 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-4npks" event={"ID":"2893cf39-ba4a-49bc-9660-c3123ae361ba","Type":"ContainerStarted","Data":"17270eb847da8bd7e7241f0b66da0c4e4bc312f811491b9268ba40991cb191d2"} Aug 04 10:33:59.564348 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:59.564216 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-4npks" Aug 04 10:33:59.565383 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:59.565359 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-gwn2m" event={"ID":"b3f3b35a-3c47-4410-afd5-671953692c16","Type":"ContainerStarted","Data":"aba046aa54bb08f26cf408fa8ed80a20bb00f087521aa1f46c558ff7f80e709a"} Aug 04 10:33:59.592399 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:33:59.592349 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5985564776-4npks" podStartSLOduration=1.246293724 podStartE2EDuration="3.592331808s" totalImagesPullingTime="2.346038084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:33:59.589731681 +0000 UTC m=+379.010619093" watchObservedRunningTime="2026-08-04 10:33:59.592331808 +0000 UTC m=+379.013219221" Aug 04 10:34:00.571389 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:00.571341 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-gwn2m" event={"ID":"b3f3b35a-3c47-4410-afd5-671953692c16","Type":"ContainerStarted","Data":"65fdbced5b17555da134e2719b17ba394ba6ce28ed14b408223dc66293856f44"} Aug 04 10:34:00.571878 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:00.571410 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d88948d8-gwn2m" Aug 04 10:34:00.593839 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:00.593781 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d88948d8-gwn2m" podStartSLOduration=1.125512351 podStartE2EDuration="2.593762513s" totalImagesPullingTime="1.468250162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:33:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:34:00.592384263 +0000 UTC m=+380.013271677" watchObservedRunningTime="2026-08-04 10:34:00.593762513 +0000 UTC m=+380.014649921" Aug 04 10:34:03.189900 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:03.189867 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-8frtz"] Aug 04 10:34:03.192372 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:03.192354 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-8frtz" Aug 04 10:34:03.195379 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:03.195360 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-spfp7\"" Aug 04 10:34:03.213971 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:03.213939 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-8frtz"] Aug 04 10:34:03.315292 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:03.315256 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6cbc65c0-d3ce-4f62-9a33-a70b54d54926-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-8frtz\" (UID: \"6cbc65c0-d3ce-4f62-9a33-a70b54d54926\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-8frtz" Aug 04 10:34:03.315484 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:03.315345 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z4kh\" (UniqueName: \"kubernetes.io/projected/6cbc65c0-d3ce-4f62-9a33-a70b54d54926-kube-api-access-5z4kh\") pod \"kuadrant-operator-controller-manager-58fc56757c-8frtz\" (UID: \"6cbc65c0-d3ce-4f62-9a33-a70b54d54926\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-8frtz" Aug 04 10:34:03.416055 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:03.416018 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6cbc65c0-d3ce-4f62-9a33-a70b54d54926-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-8frtz\" (UID: \"6cbc65c0-d3ce-4f62-9a33-a70b54d54926\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-8frtz" Aug 04 10:34:03.416222 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:03.416080 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z4kh\" (UniqueName: \"kubernetes.io/projected/6cbc65c0-d3ce-4f62-9a33-a70b54d54926-kube-api-access-5z4kh\") pod \"kuadrant-operator-controller-manager-58fc56757c-8frtz\" (UID: \"6cbc65c0-d3ce-4f62-9a33-a70b54d54926\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-8frtz" Aug 04 10:34:03.416425 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:03.416406 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6cbc65c0-d3ce-4f62-9a33-a70b54d54926-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-8frtz\" (UID: \"6cbc65c0-d3ce-4f62-9a33-a70b54d54926\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-8frtz" Aug 04 10:34:03.436967 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:03.436936 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z4kh\" (UniqueName: \"kubernetes.io/projected/6cbc65c0-d3ce-4f62-9a33-a70b54d54926-kube-api-access-5z4kh\") pod \"kuadrant-operator-controller-manager-58fc56757c-8frtz\" (UID: \"6cbc65c0-d3ce-4f62-9a33-a70b54d54926\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-8frtz" Aug 04 10:34:03.501804 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:03.501713 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-8frtz" Aug 04 10:34:03.658131 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:03.658086 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-8frtz"] Aug 04 10:34:03.661153 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:34:03.661123 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cbc65c0_d3ce_4f62_9a33_a70b54d54926.slice/crio-7bacee673d357d5071cd917a897b8cae143570f112a70cfb0e26f08025125f3d WatchSource:0}: Error finding container 7bacee673d357d5071cd917a897b8cae143570f112a70cfb0e26f08025125f3d: Status 404 returned error can't find the container with id 7bacee673d357d5071cd917a897b8cae143570f112a70cfb0e26f08025125f3d Aug 04 10:34:04.597765 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:04.597721 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-8frtz" event={"ID":"6cbc65c0-d3ce-4f62-9a33-a70b54d54926","Type":"ContainerStarted","Data":"7bacee673d357d5071cd917a897b8cae143570f112a70cfb0e26f08025125f3d"} Aug 04 10:34:05.723579 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:05.723521 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7qpw"] Aug 04 10:34:05.726833 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:05.726809 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7qpw" Aug 04 10:34:05.729974 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:05.729938 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-l98ct\"" Aug 04 10:34:05.747335 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:05.747278 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7qpw"] Aug 04 10:34:05.836963 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:05.836925 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57sg2\" (UniqueName: \"kubernetes.io/projected/e8e0f219-d282-4614-9bee-9b79c01e8f0f-kube-api-access-57sg2\") pod \"limitador-operator-controller-manager-6c549b6d85-v7qpw\" (UID: \"e8e0f219-d282-4614-9bee-9b79c01e8f0f\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7qpw" Aug 04 10:34:05.937904 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:05.937859 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57sg2\" (UniqueName: \"kubernetes.io/projected/e8e0f219-d282-4614-9bee-9b79c01e8f0f-kube-api-access-57sg2\") pod \"limitador-operator-controller-manager-6c549b6d85-v7qpw\" (UID: \"e8e0f219-d282-4614-9bee-9b79c01e8f0f\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7qpw" Aug 04 10:34:05.947925 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:05.947878 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57sg2\" (UniqueName: \"kubernetes.io/projected/e8e0f219-d282-4614-9bee-9b79c01e8f0f-kube-api-access-57sg2\") pod \"limitador-operator-controller-manager-6c549b6d85-v7qpw\" (UID: \"e8e0f219-d282-4614-9bee-9b79c01e8f0f\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7qpw" Aug 04 10:34:06.039891 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:06.039803 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7qpw" Aug 04 10:34:06.197210 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:06.197167 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7qpw"] Aug 04 10:34:07.391509 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:34:07.391480 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e0f219_d282_4614_9bee_9b79c01e8f0f.slice/crio-d5aeb7d3676337dff36dedd1cd93dac7fd219cdad0525cdb166116e24f384708 WatchSource:0}: Error finding container d5aeb7d3676337dff36dedd1cd93dac7fd219cdad0525cdb166116e24f384708: Status 404 returned error can't find the container with id d5aeb7d3676337dff36dedd1cd93dac7fd219cdad0525cdb166116e24f384708 Aug 04 10:34:07.612863 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:07.612829 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7qpw" event={"ID":"e8e0f219-d282-4614-9bee-9b79c01e8f0f","Type":"ContainerStarted","Data":"d5aeb7d3676337dff36dedd1cd93dac7fd219cdad0525cdb166116e24f384708"} Aug 04 10:34:07.615154 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:07.615120 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-8frtz" event={"ID":"6cbc65c0-d3ce-4f62-9a33-a70b54d54926","Type":"ContainerStarted","Data":"0d9d84e60c71874fde61a68e927e88a2561195b11f73be3597a0fb2814d5e500"} Aug 04 10:34:07.615900 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:07.615875 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-8frtz" Aug 04 10:34:07.641250 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:07.641203 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-8frtz" podStartSLOduration=0.846123406 podStartE2EDuration="4.641186078s" totalImagesPullingTime="3.795062672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:34:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:34:07.639671921 +0000 UTC m=+387.060559332" watchObservedRunningTime="2026-08-04 10:34:07.641186078 +0000 UTC m=+387.062073480" Aug 04 10:34:09.625737 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:09.625696 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7qpw" event={"ID":"e8e0f219-d282-4614-9bee-9b79c01e8f0f","Type":"ContainerStarted","Data":"6f1e161b1da997d5aed0a698ead35a41a37a5aa05119c5011579b3980c724a22"} Aug 04 10:34:09.626190 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:09.625786 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7qpw" Aug 04 10:34:09.646722 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:09.646675 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7qpw" podStartSLOduration=3.172294829 podStartE2EDuration="4.646659303s" totalImagesPullingTime="1.474364474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:34:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:34:09.644810719 +0000 UTC m=+389.065698132" watchObservedRunningTime="2026-08-04 10:34:09.646659303 +0000 UTC m=+389.067546714" Aug 04 10:34:10.574219 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:10.574184 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-4npks" Aug 04 10:34:11.577943 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:11.577916 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d88948d8-gwn2m" Aug 04 10:34:19.628193 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:19.628157 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-8frtz" Aug 04 10:34:20.112360 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.112329 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-855557bfff-r99l4"] Aug 04 10:34:20.115815 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.115798 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-855557bfff-r99l4" Aug 04 10:34:20.118767 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.118744 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Aug 04 10:34:20.118899 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.118791 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Aug 04 10:34:20.118899 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.118841 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Aug 04 10:34:20.120018 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.119985 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-r84x5\"" Aug 04 10:34:20.120018 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.120033 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Aug 04 10:34:20.120264 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.120034 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Aug 04 10:34:20.120264 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.120096 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Aug 04 10:34:20.120391 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.120295 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Aug 04 10:34:20.124918 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.124897 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Aug 04 10:34:20.127753 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.127733 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-855557bfff-r99l4"] Aug 04 10:34:20.153802 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.153767 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4-trusted-ca-bundle\") pod \"console-855557bfff-r99l4\" (UID: \"5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4\") " pod="openshift-console/console-855557bfff-r99l4" Aug 04 10:34:20.153802 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.153804 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4-console-serving-cert\") pod \"console-855557bfff-r99l4\" (UID: \"5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4\") " pod="openshift-console/console-855557bfff-r99l4" Aug 04 10:34:20.154027 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.153830 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4-service-ca\") pod \"console-855557bfff-r99l4\" (UID: \"5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4\") " pod="openshift-console/console-855557bfff-r99l4" Aug 04 10:34:20.154027 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.153852 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4-console-oauth-config\") pod \"console-855557bfff-r99l4\" (UID: \"5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4\") " pod="openshift-console/console-855557bfff-r99l4" Aug 04 10:34:20.154027 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.153903 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4-console-config\") pod \"console-855557bfff-r99l4\" (UID: \"5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4\") " pod="openshift-console/console-855557bfff-r99l4" Aug 04 10:34:20.154027 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.153965 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk427\" (UniqueName: \"kubernetes.io/projected/5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4-kube-api-access-hk427\") pod \"console-855557bfff-r99l4\" (UID: \"5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4\") " pod="openshift-console/console-855557bfff-r99l4" Aug 04 10:34:20.154027 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.153997 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4-oauth-serving-cert\") pod \"console-855557bfff-r99l4\" (UID: \"5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4\") " pod="openshift-console/console-855557bfff-r99l4" Aug 04 10:34:20.255439 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.255338 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4-trusted-ca-bundle\") pod \"console-855557bfff-r99l4\" (UID: \"5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4\") " pod="openshift-console/console-855557bfff-r99l4" Aug 04 10:34:20.255439 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.255410 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4-console-serving-cert\") pod \"console-855557bfff-r99l4\" (UID: \"5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4\") " pod="openshift-console/console-855557bfff-r99l4" Aug 04 10:34:20.255776 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.255457 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4-service-ca\") pod \"console-855557bfff-r99l4\" (UID: \"5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4\") " pod="openshift-console/console-855557bfff-r99l4" Aug 04 10:34:20.255776 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.255483 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4-console-oauth-config\") pod \"console-855557bfff-r99l4\" (UID: \"5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4\") " pod="openshift-console/console-855557bfff-r99l4" Aug 04 10:34:20.255776 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.255521 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4-console-config\") pod \"console-855557bfff-r99l4\" (UID: \"5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4\") " pod="openshift-console/console-855557bfff-r99l4" Aug 04 10:34:20.255776 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.255552 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk427\" (UniqueName: \"kubernetes.io/projected/5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4-kube-api-access-hk427\") pod \"console-855557bfff-r99l4\" (UID: \"5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4\") " pod="openshift-console/console-855557bfff-r99l4" Aug 04 10:34:20.255776 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.255623 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4-oauth-serving-cert\") pod \"console-855557bfff-r99l4\" (UID: \"5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4\") " pod="openshift-console/console-855557bfff-r99l4" Aug 04 10:34:20.256126 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.256092 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4-trusted-ca-bundle\") pod \"console-855557bfff-r99l4\" (UID: \"5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4\") " pod="openshift-console/console-855557bfff-r99l4" Aug 04 10:34:20.256235 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.256216 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4-service-ca\") pod \"console-855557bfff-r99l4\" (UID: \"5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4\") " pod="openshift-console/console-855557bfff-r99l4" Aug 04 10:34:20.256519 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.256502 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4-console-config\") pod \"console-855557bfff-r99l4\" (UID: \"5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4\") " pod="openshift-console/console-855557bfff-r99l4" Aug 04 10:34:20.256725 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.256703 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4-oauth-serving-cert\") pod \"console-855557bfff-r99l4\" (UID: \"5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4\") " pod="openshift-console/console-855557bfff-r99l4" Aug 04 10:34:20.258141 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.258112 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4-console-serving-cert\") pod \"console-855557bfff-r99l4\" (UID: \"5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4\") " pod="openshift-console/console-855557bfff-r99l4" Aug 04 10:34:20.258226 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.258163 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4-console-oauth-config\") pod \"console-855557bfff-r99l4\" (UID: \"5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4\") " pod="openshift-console/console-855557bfff-r99l4" Aug 04 10:34:20.273978 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.273939 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk427\" (UniqueName: \"kubernetes.io/projected/5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4-kube-api-access-hk427\") pod \"console-855557bfff-r99l4\" (UID: \"5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4\") " pod="openshift-console/console-855557bfff-r99l4" Aug 04 10:34:20.425751 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.425713 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-855557bfff-r99l4" Aug 04 10:34:20.575789 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.575760 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-855557bfff-r99l4"] Aug 04 10:34:20.578002 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:34:20.577975 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cb62afe_f3a2_4d0e_b7d6_4dcc2b77fae4.slice/crio-cd492ef360fc74978e2cfe8fe6e99d20b39ccd4e24ac7ea047417052a7714403 WatchSource:0}: Error finding container cd492ef360fc74978e2cfe8fe6e99d20b39ccd4e24ac7ea047417052a7714403: Status 404 returned error can't find the container with id cd492ef360fc74978e2cfe8fe6e99d20b39ccd4e24ac7ea047417052a7714403 Aug 04 10:34:20.632309 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.632285 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7qpw" Aug 04 10:34:20.674220 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.674180 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-855557bfff-r99l4" event={"ID":"5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4","Type":"ContainerStarted","Data":"0165690bf4798321686f9a95474cbf331650577f70ac567fe667d0ce2726cbaf"} Aug 04 10:34:20.674384 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.674232 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-855557bfff-r99l4" event={"ID":"5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4","Type":"ContainerStarted","Data":"cd492ef360fc74978e2cfe8fe6e99d20b39ccd4e24ac7ea047417052a7714403"} Aug 04 10:34:20.679645 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.679617 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-kbkjz"] Aug 04 10:34:20.682799 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.682784 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-kbkjz" Aug 04 10:34:20.703983 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.703881 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-855557bfff-r99l4" podStartSLOduration=0.703863528 podStartE2EDuration="703.863528ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:34:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:34:20.702256584 +0000 UTC m=+400.123143995" watchObservedRunningTime="2026-08-04 10:34:20.703863528 +0000 UTC m=+400.124750939" Aug 04 10:34:20.706881 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.706854 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-kbkjz"] Aug 04 10:34:20.760495 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.760413 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/af2d084c-be33-4036-9492-de972ee3b41c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-kbkjz\" (UID: \"af2d084c-be33-4036-9492-de972ee3b41c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-kbkjz" Aug 04 10:34:20.760495 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.760461 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4tg5\" (UniqueName: \"kubernetes.io/projected/af2d084c-be33-4036-9492-de972ee3b41c-kube-api-access-l4tg5\") pod \"kuadrant-operator-controller-manager-96db98c47-kbkjz\" (UID: \"af2d084c-be33-4036-9492-de972ee3b41c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-kbkjz" Aug 04 10:34:20.861755 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.861716 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/af2d084c-be33-4036-9492-de972ee3b41c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-kbkjz\" (UID: \"af2d084c-be33-4036-9492-de972ee3b41c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-kbkjz" Aug 04 10:34:20.861755 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.861759 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4tg5\" (UniqueName: \"kubernetes.io/projected/af2d084c-be33-4036-9492-de972ee3b41c-kube-api-access-l4tg5\") pod \"kuadrant-operator-controller-manager-96db98c47-kbkjz\" (UID: \"af2d084c-be33-4036-9492-de972ee3b41c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-kbkjz" Aug 04 10:34:20.862126 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.862104 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/af2d084c-be33-4036-9492-de972ee3b41c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-kbkjz\" (UID: \"af2d084c-be33-4036-9492-de972ee3b41c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-kbkjz" Aug 04 10:34:20.876767 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.876739 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4tg5\" (UniqueName: \"kubernetes.io/projected/af2d084c-be33-4036-9492-de972ee3b41c-kube-api-access-l4tg5\") pod \"kuadrant-operator-controller-manager-96db98c47-kbkjz\" (UID: \"af2d084c-be33-4036-9492-de972ee3b41c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-kbkjz" Aug 04 10:34:20.993462 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:20.993424 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-kbkjz" Aug 04 10:34:21.154420 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.154391 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-kbkjz"] Aug 04 10:34:21.157113 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:34:21.157078 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf2d084c_be33_4036_9492_de972ee3b41c.slice/crio-debeca5f1cbd6f53a221bb5d38728b5e6e41759bf438a24799c257f5c6a4857a WatchSource:0}: Error finding container debeca5f1cbd6f53a221bb5d38728b5e6e41759bf438a24799c257f5c6a4857a: Status 404 returned error can't find the container with id debeca5f1cbd6f53a221bb5d38728b5e6e41759bf438a24799c257f5c6a4857a Aug 04 10:34:21.681237 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.681198 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-kbkjz" event={"ID":"af2d084c-be33-4036-9492-de972ee3b41c","Type":"ContainerStarted","Data":"13b2f63108d19461f5c70b46d3b69f3815a97087ba19806976160e08af743c3d"} Aug 04 10:34:21.681696 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.681348 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-kbkjz" Aug 04 10:34:21.681696 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.681394 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-kbkjz" event={"ID":"af2d084c-be33-4036-9492-de972ee3b41c","Type":"ContainerStarted","Data":"debeca5f1cbd6f53a221bb5d38728b5e6e41759bf438a24799c257f5c6a4857a"} Aug 04 10:34:21.715168 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.715119 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-kbkjz" podStartSLOduration=1.715102203 podStartE2EDuration="1.715102203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:34:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:34:21.712400999 +0000 UTC m=+401.133288409" watchObservedRunningTime="2026-08-04 10:34:21.715102203 +0000 UTC m=+401.135989614" Aug 04 10:34:21.767364 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.767329 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-8frtz"] Aug 04 10:34:21.767576 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.767533 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-8frtz" podUID="6cbc65c0-d3ce-4f62-9a33-a70b54d54926" containerName="manager" containerID="cri-o://0d9d84e60c71874fde61a68e927e88a2561195b11f73be3597a0fb2814d5e500" gracePeriod=2 Aug 04 10:34:21.784080 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.784049 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-8frtz"] Aug 04 10:34:21.784756 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:34:21.784725 2580 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"6cbc65c0-d3ce-4f62-9a33-a70b54d54926\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-58fc56757c-8frtz\": pods \"kuadrant-operator-controller-manager-58fc56757c-8frtz\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-8frtz" Aug 04 10:34:21.793048 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.793019 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-54wp4"] Aug 04 10:34:21.793497 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.793473 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cbc65c0-d3ce-4f62-9a33-a70b54d54926" containerName="manager" Aug 04 10:34:21.793497 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.793497 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cbc65c0-d3ce-4f62-9a33-a70b54d54926" containerName="manager" Aug 04 10:34:21.793687 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.793618 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cbc65c0-d3ce-4f62-9a33-a70b54d54926" containerName="manager" Aug 04 10:34:21.796780 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.796759 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-kbkjz"] Aug 04 10:34:21.796903 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.796856 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-54wp4" Aug 04 10:34:21.800012 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:34:21.799973 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-8frtz\" is forbidden: User \"system:node:ip-10-0-142-113.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-113.ec2.internal' and this object" podUID="6cbc65c0-d3ce-4f62-9a33-a70b54d54926" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-8frtz" Aug 04 10:34:21.801915 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.801891 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-kbkjz"] Aug 04 10:34:21.817429 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.817388 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-54wp4"] Aug 04 10:34:21.825317 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.825285 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7qpw"] Aug 04 10:34:21.825519 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.825498 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7qpw" podUID="e8e0f219-d282-4614-9bee-9b79c01e8f0f" containerName="manager" containerID="cri-o://6f1e161b1da997d5aed0a698ead35a41a37a5aa05119c5011579b3980c724a22" gracePeriod=2 Aug 04 10:34:21.829347 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.829316 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-lkrq8"] Aug 04 10:34:21.829912 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.829882 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af2d084c-be33-4036-9492-de972ee3b41c" containerName="manager" Aug 04 10:34:21.829912 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.829906 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="af2d084c-be33-4036-9492-de972ee3b41c" containerName="manager" Aug 04 10:34:21.830160 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.829982 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="af2d084c-be33-4036-9492-de972ee3b41c" containerName="manager" Aug 04 10:34:21.833650 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.833628 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7qpw"] Aug 04 10:34:21.833790 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.833723 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-lkrq8" Aug 04 10:34:21.839802 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:34:21.839763 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-v7qpw\" is forbidden: User \"system:node:ip-10-0-142-113.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-113.ec2.internal' and this object" podUID="e8e0f219-d282-4614-9bee-9b79c01e8f0f" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7qpw" Aug 04 10:34:21.842319 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:34:21.842288 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-8frtz\" is forbidden: User \"system:node:ip-10-0-142-113.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-113.ec2.internal' and this object" podUID="6cbc65c0-d3ce-4f62-9a33-a70b54d54926" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-8frtz" Aug 04 10:34:21.853373 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.853347 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qkfqh"] Aug 04 10:34:21.853825 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.853802 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8e0f219-d282-4614-9bee-9b79c01e8f0f" containerName="manager" Aug 04 10:34:21.853825 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.853826 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8e0f219-d282-4614-9bee-9b79c01e8f0f" containerName="manager" Aug 04 10:34:21.853987 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.853893 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8e0f219-d282-4614-9bee-9b79c01e8f0f" containerName="manager" Aug 04 10:34:21.857171 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.857145 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-lkrq8"] Aug 04 10:34:21.857295 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.857175 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qkfqh" Aug 04 10:34:21.871605 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.871555 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4k8c\" (UniqueName: \"kubernetes.io/projected/351e1957-390f-415b-96e4-91b91caf1d55-kube-api-access-v4k8c\") pod \"kuadrant-operator-controller-manager-58fc56757c-54wp4\" (UID: \"351e1957-390f-415b-96e4-91b91caf1d55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-54wp4" Aug 04 10:34:21.871819 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.871621 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc3d11b5-6caa-45db-a5f2-5279d96c0423-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-lkrq8\" (UID: \"fc3d11b5-6caa-45db-a5f2-5279d96c0423\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-lkrq8" Aug 04 10:34:21.871819 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.871659 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzgz5\" (UniqueName: \"kubernetes.io/projected/fc3d11b5-6caa-45db-a5f2-5279d96c0423-kube-api-access-zzgz5\") pod \"kuadrant-operator-controller-manager-96db98c47-lkrq8\" (UID: \"fc3d11b5-6caa-45db-a5f2-5279d96c0423\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-lkrq8" Aug 04 10:34:21.871819 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.871689 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/351e1957-390f-415b-96e4-91b91caf1d55-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-54wp4\" (UID: \"351e1957-390f-415b-96e4-91b91caf1d55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-54wp4" Aug 04 10:34:21.876416 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.876375 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qkfqh"] Aug 04 10:34:21.922385 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:34:21.922348 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-v7qpw\" is forbidden: User \"system:node:ip-10-0-142-113.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-113.ec2.internal' and this object" podUID="e8e0f219-d282-4614-9bee-9b79c01e8f0f" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7qpw" Aug 04 10:34:21.924530 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:34:21.924505 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-8frtz\" is forbidden: User \"system:node:ip-10-0-142-113.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-113.ec2.internal' and this object" podUID="6cbc65c0-d3ce-4f62-9a33-a70b54d54926" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-8frtz" Aug 04 10:34:21.972576 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.972524 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7xts\" (UniqueName: \"kubernetes.io/projected/1e55781b-f662-4806-8b91-fe7ab66de387-kube-api-access-m7xts\") pod \"limitador-operator-controller-manager-6c549b6d85-qkfqh\" (UID: \"1e55781b-f662-4806-8b91-fe7ab66de387\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qkfqh" Aug 04 10:34:21.972740 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.972643 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4k8c\" (UniqueName: \"kubernetes.io/projected/351e1957-390f-415b-96e4-91b91caf1d55-kube-api-access-v4k8c\") pod \"kuadrant-operator-controller-manager-58fc56757c-54wp4\" (UID: \"351e1957-390f-415b-96e4-91b91caf1d55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-54wp4" Aug 04 10:34:21.972740 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.972676 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc3d11b5-6caa-45db-a5f2-5279d96c0423-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-lkrq8\" (UID: \"fc3d11b5-6caa-45db-a5f2-5279d96c0423\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-lkrq8" Aug 04 10:34:21.972740 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.972708 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzgz5\" (UniqueName: \"kubernetes.io/projected/fc3d11b5-6caa-45db-a5f2-5279d96c0423-kube-api-access-zzgz5\") pod \"kuadrant-operator-controller-manager-96db98c47-lkrq8\" (UID: \"fc3d11b5-6caa-45db-a5f2-5279d96c0423\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-lkrq8" Aug 04 10:34:21.972740 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.972730 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/351e1957-390f-415b-96e4-91b91caf1d55-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-54wp4\" (UID: \"351e1957-390f-415b-96e4-91b91caf1d55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-54wp4" Aug 04 10:34:21.973118 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.973102 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/351e1957-390f-415b-96e4-91b91caf1d55-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-54wp4\" (UID: \"351e1957-390f-415b-96e4-91b91caf1d55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-54wp4" Aug 04 10:34:21.973167 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.973152 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc3d11b5-6caa-45db-a5f2-5279d96c0423-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-lkrq8\" (UID: \"fc3d11b5-6caa-45db-a5f2-5279d96c0423\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-lkrq8" Aug 04 10:34:21.982986 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.982958 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4k8c\" (UniqueName: \"kubernetes.io/projected/351e1957-390f-415b-96e4-91b91caf1d55-kube-api-access-v4k8c\") pod \"kuadrant-operator-controller-manager-58fc56757c-54wp4\" (UID: \"351e1957-390f-415b-96e4-91b91caf1d55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-54wp4" Aug 04 10:34:21.984001 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:21.983980 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzgz5\" (UniqueName: \"kubernetes.io/projected/fc3d11b5-6caa-45db-a5f2-5279d96c0423-kube-api-access-zzgz5\") pod \"kuadrant-operator-controller-manager-96db98c47-lkrq8\" (UID: \"fc3d11b5-6caa-45db-a5f2-5279d96c0423\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-lkrq8" Aug 04 10:34:22.073663 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.073634 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7xts\" (UniqueName: \"kubernetes.io/projected/1e55781b-f662-4806-8b91-fe7ab66de387-kube-api-access-m7xts\") pod \"limitador-operator-controller-manager-6c549b6d85-qkfqh\" (UID: \"1e55781b-f662-4806-8b91-fe7ab66de387\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qkfqh" Aug 04 10:34:22.080996 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.080968 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-8frtz" Aug 04 10:34:22.083227 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.083204 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7xts\" (UniqueName: \"kubernetes.io/projected/1e55781b-f662-4806-8b91-fe7ab66de387-kube-api-access-m7xts\") pod \"limitador-operator-controller-manager-6c549b6d85-qkfqh\" (UID: \"1e55781b-f662-4806-8b91-fe7ab66de387\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qkfqh" Aug 04 10:34:22.083498 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:34:22.083476 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-v7qpw\" is forbidden: User \"system:node:ip-10-0-142-113.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-113.ec2.internal' and this object" podUID="e8e0f219-d282-4614-9bee-9b79c01e8f0f" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7qpw" Aug 04 10:34:22.084221 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.084206 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7qpw" Aug 04 10:34:22.085811 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:34:22.085790 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-8frtz\" is forbidden: User \"system:node:ip-10-0-142-113.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-113.ec2.internal' and this object" podUID="6cbc65c0-d3ce-4f62-9a33-a70b54d54926" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-8frtz" Aug 04 10:34:22.088054 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:34:22.088032 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-v7qpw\" is forbidden: User \"system:node:ip-10-0-142-113.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-113.ec2.internal' and this object" podUID="e8e0f219-d282-4614-9bee-9b79c01e8f0f" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7qpw" Aug 04 10:34:22.090340 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:34:22.090320 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-8frtz\" is forbidden: User \"system:node:ip-10-0-142-113.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-113.ec2.internal' and this object" podUID="6cbc65c0-d3ce-4f62-9a33-a70b54d54926" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-8frtz" Aug 04 10:34:22.174348 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.174305 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57sg2\" (UniqueName: \"kubernetes.io/projected/e8e0f219-d282-4614-9bee-9b79c01e8f0f-kube-api-access-57sg2\") pod \"e8e0f219-d282-4614-9bee-9b79c01e8f0f\" (UID: \"e8e0f219-d282-4614-9bee-9b79c01e8f0f\") " Aug 04 10:34:22.174511 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.174418 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6cbc65c0-d3ce-4f62-9a33-a70b54d54926-extensions-socket-volume\") pod \"6cbc65c0-d3ce-4f62-9a33-a70b54d54926\" (UID: \"6cbc65c0-d3ce-4f62-9a33-a70b54d54926\") " Aug 04 10:34:22.174511 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.174437 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5z4kh\" (UniqueName: \"kubernetes.io/projected/6cbc65c0-d3ce-4f62-9a33-a70b54d54926-kube-api-access-5z4kh\") pod \"6cbc65c0-d3ce-4f62-9a33-a70b54d54926\" (UID: \"6cbc65c0-d3ce-4f62-9a33-a70b54d54926\") " Aug 04 10:34:22.175085 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.174994 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cbc65c0-d3ce-4f62-9a33-a70b54d54926-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6cbc65c0-d3ce-4f62-9a33-a70b54d54926" (UID: "6cbc65c0-d3ce-4f62-9a33-a70b54d54926"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:34:22.176633 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.176610 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e0f219-d282-4614-9bee-9b79c01e8f0f-kube-api-access-57sg2" (OuterVolumeSpecName: "kube-api-access-57sg2") pod "e8e0f219-d282-4614-9bee-9b79c01e8f0f" (UID: "e8e0f219-d282-4614-9bee-9b79c01e8f0f"). InnerVolumeSpecName "kube-api-access-57sg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:34:22.176715 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.176680 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cbc65c0-d3ce-4f62-9a33-a70b54d54926-kube-api-access-5z4kh" (OuterVolumeSpecName: "kube-api-access-5z4kh") pod "6cbc65c0-d3ce-4f62-9a33-a70b54d54926" (UID: "6cbc65c0-d3ce-4f62-9a33-a70b54d54926"). InnerVolumeSpecName "kube-api-access-5z4kh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:34:22.198913 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.198829 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-54wp4" Aug 04 10:34:22.205904 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.205877 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-lkrq8" Aug 04 10:34:22.212027 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.212000 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qkfqh" Aug 04 10:34:22.277514 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.276161 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6cbc65c0-d3ce-4f62-9a33-a70b54d54926-extensions-socket-volume\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:34:22.277514 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.276195 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z4kh\" (UniqueName: \"kubernetes.io/projected/6cbc65c0-d3ce-4f62-9a33-a70b54d54926-kube-api-access-5z4kh\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:34:22.277514 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.276211 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57sg2\" (UniqueName: \"kubernetes.io/projected/e8e0f219-d282-4614-9bee-9b79c01e8f0f-kube-api-access-57sg2\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:34:22.368942 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.368909 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-lkrq8"] Aug 04 10:34:22.373716 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:34:22.373690 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3d11b5_6caa_45db_a5f2_5279d96c0423.slice/crio-064668f386d65dfb7a5d13338408d0d759af51aa5a459b5a0e23d1902a4db526 WatchSource:0}: Error finding container 064668f386d65dfb7a5d13338408d0d759af51aa5a459b5a0e23d1902a4db526: Status 404 returned error can't find the container with id 064668f386d65dfb7a5d13338408d0d759af51aa5a459b5a0e23d1902a4db526 Aug 04 10:34:22.390521 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.390495 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-54wp4"] Aug 04 10:34:22.392357 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:34:22.392328 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod351e1957_390f_415b_96e4_91b91caf1d55.slice/crio-23aa89751200b468ae025a672afcfecc90422a8c2f806e740ddc251294dc34c1 WatchSource:0}: Error finding container 23aa89751200b468ae025a672afcfecc90422a8c2f806e740ddc251294dc34c1: Status 404 returned error can't find the container with id 23aa89751200b468ae025a672afcfecc90422a8c2f806e740ddc251294dc34c1 Aug 04 10:34:22.408482 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.408392 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qkfqh"] Aug 04 10:34:22.412857 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:34:22.412821 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e55781b_f662_4806_8b91_fe7ab66de387.slice/crio-d4cef414bc3fff5c2afef8f9c4cd47150e886c8ff78c549d423346416b788d78 WatchSource:0}: Error finding container d4cef414bc3fff5c2afef8f9c4cd47150e886c8ff78c549d423346416b788d78: Status 404 returned error can't find the container with id d4cef414bc3fff5c2afef8f9c4cd47150e886c8ff78c549d423346416b788d78 Aug 04 10:34:22.696065 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.696022 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-54wp4" event={"ID":"351e1957-390f-415b-96e4-91b91caf1d55","Type":"ContainerStarted","Data":"5b0704883d77d02af53cec586d8482d706ee5759a723c3a58b09081ad559b3c3"} Aug 04 10:34:22.696527 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.696126 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-54wp4" event={"ID":"351e1957-390f-415b-96e4-91b91caf1d55","Type":"ContainerStarted","Data":"23aa89751200b468ae025a672afcfecc90422a8c2f806e740ddc251294dc34c1"} Aug 04 10:34:22.696527 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.696158 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-54wp4" Aug 04 10:34:22.697520 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.697496 2580 generic.go:356] "Generic (PLEG): container finished" podID="6cbc65c0-d3ce-4f62-9a33-a70b54d54926" containerID="0d9d84e60c71874fde61a68e927e88a2561195b11f73be3597a0fb2814d5e500" exitCode=0 Aug 04 10:34:22.697642 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.697555 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-8frtz" Aug 04 10:34:22.697642 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.697608 2580 scope.go:117] "RemoveContainer" containerID="0d9d84e60c71874fde61a68e927e88a2561195b11f73be3597a0fb2814d5e500" Aug 04 10:34:22.699779 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.699754 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-lkrq8" event={"ID":"fc3d11b5-6caa-45db-a5f2-5279d96c0423","Type":"ContainerStarted","Data":"5b8cd92c762297b96902fe11caac9189e96efa297a0cca2cd16988ff97984d2f"} Aug 04 10:34:22.699928 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.699812 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-lkrq8" Aug 04 10:34:22.699928 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.699839 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-lkrq8" event={"ID":"fc3d11b5-6caa-45db-a5f2-5279d96c0423","Type":"ContainerStarted","Data":"064668f386d65dfb7a5d13338408d0d759af51aa5a459b5a0e23d1902a4db526"} Aug 04 10:34:22.701904 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.701876 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qkfqh" event={"ID":"1e55781b-f662-4806-8b91-fe7ab66de387","Type":"ContainerStarted","Data":"c4dd878119183db3bebcee62afde482a8639e819473d06712b69514ee18c8a16"} Aug 04 10:34:22.702043 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.701923 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qkfqh" event={"ID":"1e55781b-f662-4806-8b91-fe7ab66de387","Type":"ContainerStarted","Data":"d4cef414bc3fff5c2afef8f9c4cd47150e886c8ff78c549d423346416b788d78"} Aug 04 10:34:22.702043 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.702005 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qkfqh" Aug 04 10:34:22.703605 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.703517 2580 generic.go:356] "Generic (PLEG): container finished" podID="e8e0f219-d282-4614-9bee-9b79c01e8f0f" containerID="6f1e161b1da997d5aed0a698ead35a41a37a5aa05119c5011579b3980c724a22" exitCode=0 Aug 04 10:34:22.703715 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.703611 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7qpw" Aug 04 10:34:22.703961 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:34:22.703939 2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-kbkjz" container="manager" Aug 04 10:34:22.708831 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.708812 2580 scope.go:117] "RemoveContainer" containerID="0d9d84e60c71874fde61a68e927e88a2561195b11f73be3597a0fb2814d5e500" Aug 04 10:34:22.709139 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:34:22.709115 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d9d84e60c71874fde61a68e927e88a2561195b11f73be3597a0fb2814d5e500\": container with ID starting with 0d9d84e60c71874fde61a68e927e88a2561195b11f73be3597a0fb2814d5e500 not found: ID does not exist" containerID="0d9d84e60c71874fde61a68e927e88a2561195b11f73be3597a0fb2814d5e500" Aug 04 10:34:22.709227 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.709147 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d9d84e60c71874fde61a68e927e88a2561195b11f73be3597a0fb2814d5e500"} err="failed to get container status \"0d9d84e60c71874fde61a68e927e88a2561195b11f73be3597a0fb2814d5e500\": rpc error: code = NotFound desc = could not find container \"0d9d84e60c71874fde61a68e927e88a2561195b11f73be3597a0fb2814d5e500\": container with ID starting with 0d9d84e60c71874fde61a68e927e88a2561195b11f73be3597a0fb2814d5e500 not found: ID does not exist" Aug 04 10:34:22.709227 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.709167 2580 scope.go:117] "RemoveContainer" containerID="6f1e161b1da997d5aed0a698ead35a41a37a5aa05119c5011579b3980c724a22" Aug 04 10:34:22.718735 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.718716 2580 scope.go:117] "RemoveContainer" containerID="6f1e161b1da997d5aed0a698ead35a41a37a5aa05119c5011579b3980c724a22" Aug 04 10:34:22.719044 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:34:22.719025 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f1e161b1da997d5aed0a698ead35a41a37a5aa05119c5011579b3980c724a22\": container with ID starting with 6f1e161b1da997d5aed0a698ead35a41a37a5aa05119c5011579b3980c724a22 not found: ID does not exist" containerID="6f1e161b1da997d5aed0a698ead35a41a37a5aa05119c5011579b3980c724a22" Aug 04 10:34:22.719113 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.719052 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f1e161b1da997d5aed0a698ead35a41a37a5aa05119c5011579b3980c724a22"} err="failed to get container status \"6f1e161b1da997d5aed0a698ead35a41a37a5aa05119c5011579b3980c724a22\": rpc error: code = NotFound desc = could not find container \"6f1e161b1da997d5aed0a698ead35a41a37a5aa05119c5011579b3980c724a22\": container with ID starting with 6f1e161b1da997d5aed0a698ead35a41a37a5aa05119c5011579b3980c724a22 not found: ID does not exist" Aug 04 10:34:22.733361 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.733316 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-54wp4" podStartSLOduration=1.733297343 podStartE2EDuration="1.733297343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:34:22.731038615 +0000 UTC m=+402.151926026" watchObservedRunningTime="2026-08-04 10:34:22.733297343 +0000 UTC m=+402.154184755" Aug 04 10:34:22.733906 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:34:22.733873 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-v7qpw\" is forbidden: User \"system:node:ip-10-0-142-113.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-113.ec2.internal' and this object" podUID="e8e0f219-d282-4614-9bee-9b79c01e8f0f" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7qpw" Aug 04 10:34:22.736203 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:34:22.736183 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-8frtz\" is forbidden: User \"system:node:ip-10-0-142-113.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-113.ec2.internal' and this object" podUID="6cbc65c0-d3ce-4f62-9a33-a70b54d54926" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-8frtz" Aug 04 10:34:22.738358 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:34:22.738335 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-8frtz\" is forbidden: User \"system:node:ip-10-0-142-113.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-113.ec2.internal' and this object" podUID="6cbc65c0-d3ce-4f62-9a33-a70b54d54926" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-8frtz" Aug 04 10:34:22.740491 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:34:22.740472 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-kbkjz\" is forbidden: User \"system:node:ip-10-0-142-113.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-113.ec2.internal' and this object" podUID="af2d084c-be33-4036-9492-de972ee3b41c" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-kbkjz" Aug 04 10:34:22.801630 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.801577 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-lkrq8" podStartSLOduration=1.801547444 podStartE2EDuration="1.801547444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:34:22.799525131 +0000 UTC m=+402.220412541" watchObservedRunningTime="2026-08-04 10:34:22.801547444 +0000 UTC m=+402.222434854" Aug 04 10:34:22.827192 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:22.827135 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qkfqh" podStartSLOduration=1.827117067 podStartE2EDuration="1.827117067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:34:22.825451062 +0000 UTC m=+402.246338472" watchObservedRunningTime="2026-08-04 10:34:22.827117067 +0000 UTC m=+402.248004477" Aug 04 10:34:22.828223 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:34:22.828193 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-v7qpw\" is forbidden: User \"system:node:ip-10-0-142-113.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-113.ec2.internal' and this object" podUID="e8e0f219-d282-4614-9bee-9b79c01e8f0f" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7qpw" Aug 04 10:34:23.100628 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:23.100535 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cbc65c0-d3ce-4f62-9a33-a70b54d54926" path="/var/lib/kubelet/pods/6cbc65c0-d3ce-4f62-9a33-a70b54d54926/volumes" Aug 04 10:34:23.100953 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:23.100934 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8e0f219-d282-4614-9bee-9b79c01e8f0f" path="/var/lib/kubelet/pods/e8e0f219-d282-4614-9bee-9b79c01e8f0f/volumes" Aug 04 10:34:23.711151 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:23.711117 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-kbkjz" podUID="af2d084c-be33-4036-9492-de972ee3b41c" containerName="manager" containerID="cri-o://13b2f63108d19461f5c70b46d3b69f3815a97087ba19806976160e08af743c3d" gracePeriod=2 Aug 04 10:34:23.957540 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:23.957512 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-kbkjz" Aug 04 10:34:23.993064 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:23.992981 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4tg5\" (UniqueName: \"kubernetes.io/projected/af2d084c-be33-4036-9492-de972ee3b41c-kube-api-access-l4tg5\") pod \"af2d084c-be33-4036-9492-de972ee3b41c\" (UID: \"af2d084c-be33-4036-9492-de972ee3b41c\") " Aug 04 10:34:23.993064 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:23.993020 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/af2d084c-be33-4036-9492-de972ee3b41c-extensions-socket-volume\") pod \"af2d084c-be33-4036-9492-de972ee3b41c\" (UID: \"af2d084c-be33-4036-9492-de972ee3b41c\") " Aug 04 10:34:23.993418 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:23.993387 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af2d084c-be33-4036-9492-de972ee3b41c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "af2d084c-be33-4036-9492-de972ee3b41c" (UID: "af2d084c-be33-4036-9492-de972ee3b41c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:34:23.995227 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:23.995209 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af2d084c-be33-4036-9492-de972ee3b41c-kube-api-access-l4tg5" (OuterVolumeSpecName: "kube-api-access-l4tg5") pod "af2d084c-be33-4036-9492-de972ee3b41c" (UID: "af2d084c-be33-4036-9492-de972ee3b41c"). InnerVolumeSpecName "kube-api-access-l4tg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:34:24.094623 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:24.094590 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4tg5\" (UniqueName: \"kubernetes.io/projected/af2d084c-be33-4036-9492-de972ee3b41c-kube-api-access-l4tg5\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:34:24.094623 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:24.094618 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/af2d084c-be33-4036-9492-de972ee3b41c-extensions-socket-volume\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:34:24.717540 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:24.717500 2580 generic.go:356] "Generic (PLEG): container finished" podID="af2d084c-be33-4036-9492-de972ee3b41c" containerID="13b2f63108d19461f5c70b46d3b69f3815a97087ba19806976160e08af743c3d" exitCode=0 Aug 04 10:34:24.717540 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:24.717543 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-kbkjz" Aug 04 10:34:24.718081 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:24.717609 2580 scope.go:117] "RemoveContainer" containerID="13b2f63108d19461f5c70b46d3b69f3815a97087ba19806976160e08af743c3d" Aug 04 10:34:24.721830 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:34:24.721798 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-kbkjz\" is forbidden: User \"system:node:ip-10-0-142-113.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-113.ec2.internal' and this object" podUID="af2d084c-be33-4036-9492-de972ee3b41c" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-kbkjz" Aug 04 10:34:24.726648 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:24.726626 2580 scope.go:117] "RemoveContainer" containerID="13b2f63108d19461f5c70b46d3b69f3815a97087ba19806976160e08af743c3d" Aug 04 10:34:24.726978 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:34:24.726956 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13b2f63108d19461f5c70b46d3b69f3815a97087ba19806976160e08af743c3d\": container with ID starting with 13b2f63108d19461f5c70b46d3b69f3815a97087ba19806976160e08af743c3d not found: ID does not exist" containerID="13b2f63108d19461f5c70b46d3b69f3815a97087ba19806976160e08af743c3d" Aug 04 10:34:24.727032 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:24.726988 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13b2f63108d19461f5c70b46d3b69f3815a97087ba19806976160e08af743c3d"} err="failed to get container status \"13b2f63108d19461f5c70b46d3b69f3815a97087ba19806976160e08af743c3d\": rpc error: code = NotFound desc = could not find container \"13b2f63108d19461f5c70b46d3b69f3815a97087ba19806976160e08af743c3d\": container with ID starting with 13b2f63108d19461f5c70b46d3b69f3815a97087ba19806976160e08af743c3d not found: ID does not exist" Aug 04 10:34:24.728222 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:34:24.728202 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-kbkjz\" is forbidden: User \"system:node:ip-10-0-142-113.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-113.ec2.internal' and this object" podUID="af2d084c-be33-4036-9492-de972ee3b41c" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-kbkjz" Aug 04 10:34:25.100114 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:25.100035 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af2d084c-be33-4036-9492-de972ee3b41c" path="/var/lib/kubelet/pods/af2d084c-be33-4036-9492-de972ee3b41c/volumes" Aug 04 10:34:30.426789 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:30.426742 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-855557bfff-r99l4" Aug 04 10:34:30.427268 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:30.426805 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-855557bfff-r99l4" Aug 04 10:34:30.431834 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:30.431809 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-855557bfff-r99l4" Aug 04 10:34:30.751833 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:30.751756 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-855557bfff-r99l4" Aug 04 10:34:33.713431 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:33.713390 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qkfqh" Aug 04 10:34:33.713933 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:33.713488 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-lkrq8" Aug 04 10:34:33.713933 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:33.713536 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-54wp4" Aug 04 10:34:33.820659 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:33.820585 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-54wp4"] Aug 04 10:34:33.820900 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:33.820868 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-54wp4" podUID="351e1957-390f-415b-96e4-91b91caf1d55" containerName="manager" containerID="cri-o://5b0704883d77d02af53cec586d8482d706ee5759a723c3a58b09081ad559b3c3" gracePeriod=10 Aug 04 10:34:34.062875 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.062849 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-54wp4" Aug 04 10:34:34.178265 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.178227 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4k8c\" (UniqueName: \"kubernetes.io/projected/351e1957-390f-415b-96e4-91b91caf1d55-kube-api-access-v4k8c\") pod \"351e1957-390f-415b-96e4-91b91caf1d55\" (UID: \"351e1957-390f-415b-96e4-91b91caf1d55\") " Aug 04 10:34:34.178426 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.178320 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/351e1957-390f-415b-96e4-91b91caf1d55-extensions-socket-volume\") pod \"351e1957-390f-415b-96e4-91b91caf1d55\" (UID: \"351e1957-390f-415b-96e4-91b91caf1d55\") " Aug 04 10:34:34.178835 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.178739 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351e1957-390f-415b-96e4-91b91caf1d55-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "351e1957-390f-415b-96e4-91b91caf1d55" (UID: "351e1957-390f-415b-96e4-91b91caf1d55"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:34:34.180466 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.180447 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351e1957-390f-415b-96e4-91b91caf1d55-kube-api-access-v4k8c" (OuterVolumeSpecName: "kube-api-access-v4k8c") pod "351e1957-390f-415b-96e4-91b91caf1d55" (UID: "351e1957-390f-415b-96e4-91b91caf1d55"). InnerVolumeSpecName "kube-api-access-v4k8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:34:34.185676 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.185650 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-4drcn"] Aug 04 10:34:34.186026 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.186010 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="351e1957-390f-415b-96e4-91b91caf1d55" containerName="manager" Aug 04 10:34:34.186026 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.186028 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="351e1957-390f-415b-96e4-91b91caf1d55" containerName="manager" Aug 04 10:34:34.186129 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.186089 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="351e1957-390f-415b-96e4-91b91caf1d55" containerName="manager" Aug 04 10:34:34.188134 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.188119 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-4drcn" Aug 04 10:34:34.209513 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.209478 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-4drcn"] Aug 04 10:34:34.279831 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.279741 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvlz2\" (UniqueName: \"kubernetes.io/projected/af82c5b5-69ad-4007-84bf-fb7380b9577e-kube-api-access-nvlz2\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-4drcn\" (UID: \"af82c5b5-69ad-4007-84bf-fb7380b9577e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-4drcn" Aug 04 10:34:34.279831 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.279801 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/af82c5b5-69ad-4007-84bf-fb7380b9577e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-4drcn\" (UID: \"af82c5b5-69ad-4007-84bf-fb7380b9577e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-4drcn" Aug 04 10:34:34.280017 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.279879 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/351e1957-390f-415b-96e4-91b91caf1d55-extensions-socket-volume\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:34:34.280017 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.279900 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4k8c\" (UniqueName: \"kubernetes.io/projected/351e1957-390f-415b-96e4-91b91caf1d55-kube-api-access-v4k8c\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:34:34.381023 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.380981 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/af82c5b5-69ad-4007-84bf-fb7380b9577e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-4drcn\" (UID: \"af82c5b5-69ad-4007-84bf-fb7380b9577e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-4drcn" Aug 04 10:34:34.381216 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.381047 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvlz2\" (UniqueName: \"kubernetes.io/projected/af82c5b5-69ad-4007-84bf-fb7380b9577e-kube-api-access-nvlz2\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-4drcn\" (UID: \"af82c5b5-69ad-4007-84bf-fb7380b9577e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-4drcn" Aug 04 10:34:34.381449 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.381424 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/af82c5b5-69ad-4007-84bf-fb7380b9577e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-4drcn\" (UID: \"af82c5b5-69ad-4007-84bf-fb7380b9577e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-4drcn" Aug 04 10:34:34.390752 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.390720 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvlz2\" (UniqueName: \"kubernetes.io/projected/af82c5b5-69ad-4007-84bf-fb7380b9577e-kube-api-access-nvlz2\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-4drcn\" (UID: \"af82c5b5-69ad-4007-84bf-fb7380b9577e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-4drcn" Aug 04 10:34:34.498406 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.498370 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-4drcn" Aug 04 10:34:34.648161 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.648134 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-4drcn"] Aug 04 10:34:34.650994 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:34:34.650965 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf82c5b5_69ad_4007_84bf_fb7380b9577e.slice/crio-0302e598e7b3080f92a5ac60937f1d3635df03bd38964c05c8f6cfa37442c2f8 WatchSource:0}: Error finding container 0302e598e7b3080f92a5ac60937f1d3635df03bd38964c05c8f6cfa37442c2f8: Status 404 returned error can't find the container with id 0302e598e7b3080f92a5ac60937f1d3635df03bd38964c05c8f6cfa37442c2f8 Aug 04 10:34:34.767529 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.767491 2580 generic.go:356] "Generic (PLEG): container finished" podID="351e1957-390f-415b-96e4-91b91caf1d55" containerID="5b0704883d77d02af53cec586d8482d706ee5759a723c3a58b09081ad559b3c3" exitCode=0 Aug 04 10:34:34.768008 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.767541 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-54wp4" Aug 04 10:34:34.768008 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.767583 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-54wp4" event={"ID":"351e1957-390f-415b-96e4-91b91caf1d55","Type":"ContainerDied","Data":"5b0704883d77d02af53cec586d8482d706ee5759a723c3a58b09081ad559b3c3"} Aug 04 10:34:34.768008 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.767639 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-54wp4" event={"ID":"351e1957-390f-415b-96e4-91b91caf1d55","Type":"ContainerDied","Data":"23aa89751200b468ae025a672afcfecc90422a8c2f806e740ddc251294dc34c1"} Aug 04 10:34:34.768008 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.767662 2580 scope.go:117] "RemoveContainer" containerID="5b0704883d77d02af53cec586d8482d706ee5759a723c3a58b09081ad559b3c3" Aug 04 10:34:34.769700 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.769673 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-4drcn" event={"ID":"af82c5b5-69ad-4007-84bf-fb7380b9577e","Type":"ContainerStarted","Data":"b8d97c3c5ff4dc1677d9889e6a62d22af769574f97e1eb2e623e7da3478c9bfa"} Aug 04 10:34:34.769831 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.769722 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-4drcn" event={"ID":"af82c5b5-69ad-4007-84bf-fb7380b9577e","Type":"ContainerStarted","Data":"0302e598e7b3080f92a5ac60937f1d3635df03bd38964c05c8f6cfa37442c2f8"} Aug 04 10:34:34.769831 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.769803 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-4drcn" Aug 04 10:34:34.776976 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.776957 2580 scope.go:117] "RemoveContainer" containerID="5b0704883d77d02af53cec586d8482d706ee5759a723c3a58b09081ad559b3c3" Aug 04 10:34:34.777327 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:34:34.777298 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b0704883d77d02af53cec586d8482d706ee5759a723c3a58b09081ad559b3c3\": container with ID starting with 5b0704883d77d02af53cec586d8482d706ee5759a723c3a58b09081ad559b3c3 not found: ID does not exist" containerID="5b0704883d77d02af53cec586d8482d706ee5759a723c3a58b09081ad559b3c3" Aug 04 10:34:34.777435 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.777335 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b0704883d77d02af53cec586d8482d706ee5759a723c3a58b09081ad559b3c3"} err="failed to get container status \"5b0704883d77d02af53cec586d8482d706ee5759a723c3a58b09081ad559b3c3\": rpc error: code = NotFound desc = could not find container \"5b0704883d77d02af53cec586d8482d706ee5759a723c3a58b09081ad559b3c3\": container with ID starting with 5b0704883d77d02af53cec586d8482d706ee5759a723c3a58b09081ad559b3c3 not found: ID does not exist" Aug 04 10:34:34.804658 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.804537 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-4drcn" podStartSLOduration=0.80452079 podStartE2EDuration="804.52079ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:34:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:34:34.803846603 +0000 UTC m=+414.224734089" watchObservedRunningTime="2026-08-04 10:34:34.80452079 +0000 UTC m=+414.225408201" Aug 04 10:34:34.821828 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.821795 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-54wp4"] Aug 04 10:34:34.825887 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:34.825861 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-54wp4"] Aug 04 10:34:35.099311 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:35.099237 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="351e1957-390f-415b-96e4-91b91caf1d55" path="/var/lib/kubelet/pods/351e1957-390f-415b-96e4-91b91caf1d55/volumes" Aug 04 10:34:45.784951 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:45.784919 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-4drcn" Aug 04 10:34:45.844840 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:45.844805 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-lkrq8"] Aug 04 10:34:45.845087 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:45.845044 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-lkrq8" podUID="fc3d11b5-6caa-45db-a5f2-5279d96c0423" containerName="manager" containerID="cri-o://5b8cd92c762297b96902fe11caac9189e96efa297a0cca2cd16988ff97984d2f" gracePeriod=10 Aug 04 10:34:46.093237 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:46.093214 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-lkrq8" Aug 04 10:34:46.188091 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:46.188060 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzgz5\" (UniqueName: \"kubernetes.io/projected/fc3d11b5-6caa-45db-a5f2-5279d96c0423-kube-api-access-zzgz5\") pod \"fc3d11b5-6caa-45db-a5f2-5279d96c0423\" (UID: \"fc3d11b5-6caa-45db-a5f2-5279d96c0423\") " Aug 04 10:34:46.188258 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:46.188101 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc3d11b5-6caa-45db-a5f2-5279d96c0423-extensions-socket-volume\") pod \"fc3d11b5-6caa-45db-a5f2-5279d96c0423\" (UID: \"fc3d11b5-6caa-45db-a5f2-5279d96c0423\") " Aug 04 10:34:46.188525 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:46.188499 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc3d11b5-6caa-45db-a5f2-5279d96c0423-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fc3d11b5-6caa-45db-a5f2-5279d96c0423" (UID: "fc3d11b5-6caa-45db-a5f2-5279d96c0423"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:34:46.190353 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:46.190327 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc3d11b5-6caa-45db-a5f2-5279d96c0423-kube-api-access-zzgz5" (OuterVolumeSpecName: "kube-api-access-zzgz5") pod "fc3d11b5-6caa-45db-a5f2-5279d96c0423" (UID: "fc3d11b5-6caa-45db-a5f2-5279d96c0423"). InnerVolumeSpecName "kube-api-access-zzgz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:34:46.288617 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:46.288583 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzgz5\" (UniqueName: \"kubernetes.io/projected/fc3d11b5-6caa-45db-a5f2-5279d96c0423-kube-api-access-zzgz5\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:34:46.288617 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:46.288610 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc3d11b5-6caa-45db-a5f2-5279d96c0423-extensions-socket-volume\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:34:46.832698 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:46.832664 2580 generic.go:356] "Generic (PLEG): container finished" podID="fc3d11b5-6caa-45db-a5f2-5279d96c0423" containerID="5b8cd92c762297b96902fe11caac9189e96efa297a0cca2cd16988ff97984d2f" exitCode=0 Aug 04 10:34:46.833100 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:46.832711 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-lkrq8" Aug 04 10:34:46.833100 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:46.832749 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-lkrq8" event={"ID":"fc3d11b5-6caa-45db-a5f2-5279d96c0423","Type":"ContainerDied","Data":"5b8cd92c762297b96902fe11caac9189e96efa297a0cca2cd16988ff97984d2f"} Aug 04 10:34:46.833100 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:46.832803 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-lkrq8" event={"ID":"fc3d11b5-6caa-45db-a5f2-5279d96c0423","Type":"ContainerDied","Data":"064668f386d65dfb7a5d13338408d0d759af51aa5a459b5a0e23d1902a4db526"} Aug 04 10:34:46.833100 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:46.832829 2580 scope.go:117] "RemoveContainer" containerID="5b8cd92c762297b96902fe11caac9189e96efa297a0cca2cd16988ff97984d2f" Aug 04 10:34:46.841968 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:46.841951 2580 scope.go:117] "RemoveContainer" containerID="5b8cd92c762297b96902fe11caac9189e96efa297a0cca2cd16988ff97984d2f" Aug 04 10:34:46.842215 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:34:46.842198 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b8cd92c762297b96902fe11caac9189e96efa297a0cca2cd16988ff97984d2f\": container with ID starting with 5b8cd92c762297b96902fe11caac9189e96efa297a0cca2cd16988ff97984d2f not found: ID does not exist" containerID="5b8cd92c762297b96902fe11caac9189e96efa297a0cca2cd16988ff97984d2f" Aug 04 10:34:46.842257 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:46.842225 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8cd92c762297b96902fe11caac9189e96efa297a0cca2cd16988ff97984d2f"} err="failed to get container status \"5b8cd92c762297b96902fe11caac9189e96efa297a0cca2cd16988ff97984d2f\": rpc error: code = NotFound desc = could not find container \"5b8cd92c762297b96902fe11caac9189e96efa297a0cca2cd16988ff97984d2f\": container with ID starting with 5b8cd92c762297b96902fe11caac9189e96efa297a0cca2cd16988ff97984d2f not found: ID does not exist" Aug 04 10:34:46.856207 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:46.856133 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-lkrq8"] Aug 04 10:34:46.870949 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:46.870918 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-lkrq8"] Aug 04 10:34:47.099095 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:47.099019 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc3d11b5-6caa-45db-a5f2-5279d96c0423" path="/var/lib/kubelet/pods/fc3d11b5-6caa-45db-a5f2-5279d96c0423/volumes" Aug 04 10:34:50.254536 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.254498 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s"] Aug 04 10:34:50.255095 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.254826 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc3d11b5-6caa-45db-a5f2-5279d96c0423" containerName="manager" Aug 04 10:34:50.255095 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.254838 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc3d11b5-6caa-45db-a5f2-5279d96c0423" containerName="manager" Aug 04 10:34:50.255095 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.254891 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc3d11b5-6caa-45db-a5f2-5279d96c0423" containerName="manager" Aug 04 10:34:50.257462 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.257445 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:34:50.260418 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.260400 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-sfrxl\"" Aug 04 10:34:50.274386 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.274356 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s"] Aug 04 10:34:50.421952 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.421912 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qrvm\" (UniqueName: \"kubernetes.io/projected/f93fb736-f6f2-4b03-9821-4036c36e2f72-kube-api-access-9qrvm\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tqq7s\" (UID: \"f93fb736-f6f2-4b03-9821-4036c36e2f72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:34:50.421952 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.421955 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93fb736-f6f2-4b03-9821-4036c36e2f72-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tqq7s\" (UID: \"f93fb736-f6f2-4b03-9821-4036c36e2f72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:34:50.422160 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.421982 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f93fb736-f6f2-4b03-9821-4036c36e2f72-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tqq7s\" (UID: \"f93fb736-f6f2-4b03-9821-4036c36e2f72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:34:50.422160 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.422064 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93fb736-f6f2-4b03-9821-4036c36e2f72-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tqq7s\" (UID: \"f93fb736-f6f2-4b03-9821-4036c36e2f72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:34:50.422160 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.422089 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f93fb736-f6f2-4b03-9821-4036c36e2f72-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tqq7s\" (UID: \"f93fb736-f6f2-4b03-9821-4036c36e2f72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:34:50.422160 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.422107 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f93fb736-f6f2-4b03-9821-4036c36e2f72-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tqq7s\" (UID: \"f93fb736-f6f2-4b03-9821-4036c36e2f72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:34:50.422160 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.422122 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f93fb736-f6f2-4b03-9821-4036c36e2f72-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tqq7s\" (UID: \"f93fb736-f6f2-4b03-9821-4036c36e2f72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:34:50.422160 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.422138 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f93fb736-f6f2-4b03-9821-4036c36e2f72-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tqq7s\" (UID: \"f93fb736-f6f2-4b03-9821-4036c36e2f72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:34:50.422160 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.422160 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f93fb736-f6f2-4b03-9821-4036c36e2f72-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tqq7s\" (UID: \"f93fb736-f6f2-4b03-9821-4036c36e2f72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:34:50.522925 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.522846 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93fb736-f6f2-4b03-9821-4036c36e2f72-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tqq7s\" (UID: \"f93fb736-f6f2-4b03-9821-4036c36e2f72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:34:50.522925 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.522877 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f93fb736-f6f2-4b03-9821-4036c36e2f72-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tqq7s\" (UID: \"f93fb736-f6f2-4b03-9821-4036c36e2f72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:34:50.522925 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.522895 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f93fb736-f6f2-4b03-9821-4036c36e2f72-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tqq7s\" (UID: \"f93fb736-f6f2-4b03-9821-4036c36e2f72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:34:50.522925 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.522911 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f93fb736-f6f2-4b03-9821-4036c36e2f72-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tqq7s\" (UID: \"f93fb736-f6f2-4b03-9821-4036c36e2f72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:34:50.523254 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.522935 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f93fb736-f6f2-4b03-9821-4036c36e2f72-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tqq7s\" (UID: \"f93fb736-f6f2-4b03-9821-4036c36e2f72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:34:50.523254 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.522972 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f93fb736-f6f2-4b03-9821-4036c36e2f72-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tqq7s\" (UID: \"f93fb736-f6f2-4b03-9821-4036c36e2f72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:34:50.523254 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.523009 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qrvm\" (UniqueName: \"kubernetes.io/projected/f93fb736-f6f2-4b03-9821-4036c36e2f72-kube-api-access-9qrvm\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tqq7s\" (UID: \"f93fb736-f6f2-4b03-9821-4036c36e2f72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:34:50.523254 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.523043 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93fb736-f6f2-4b03-9821-4036c36e2f72-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tqq7s\" (UID: \"f93fb736-f6f2-4b03-9821-4036c36e2f72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:34:50.523254 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.523068 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f93fb736-f6f2-4b03-9821-4036c36e2f72-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tqq7s\" (UID: \"f93fb736-f6f2-4b03-9821-4036c36e2f72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:34:50.523483 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.523292 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93fb736-f6f2-4b03-9821-4036c36e2f72-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tqq7s\" (UID: \"f93fb736-f6f2-4b03-9821-4036c36e2f72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:34:50.523483 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.523305 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f93fb736-f6f2-4b03-9821-4036c36e2f72-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tqq7s\" (UID: \"f93fb736-f6f2-4b03-9821-4036c36e2f72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:34:50.523638 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.523610 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93fb736-f6f2-4b03-9821-4036c36e2f72-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tqq7s\" (UID: \"f93fb736-f6f2-4b03-9821-4036c36e2f72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:34:50.523753 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.523638 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f93fb736-f6f2-4b03-9821-4036c36e2f72-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tqq7s\" (UID: \"f93fb736-f6f2-4b03-9821-4036c36e2f72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:34:50.523805 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.523787 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f93fb736-f6f2-4b03-9821-4036c36e2f72-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tqq7s\" (UID: \"f93fb736-f6f2-4b03-9821-4036c36e2f72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:34:50.525906 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.525882 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f93fb736-f6f2-4b03-9821-4036c36e2f72-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tqq7s\" (UID: \"f93fb736-f6f2-4b03-9821-4036c36e2f72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:34:50.526087 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.526067 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f93fb736-f6f2-4b03-9821-4036c36e2f72-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tqq7s\" (UID: \"f93fb736-f6f2-4b03-9821-4036c36e2f72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:34:50.531796 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.531766 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qrvm\" (UniqueName: \"kubernetes.io/projected/f93fb736-f6f2-4b03-9821-4036c36e2f72-kube-api-access-9qrvm\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tqq7s\" (UID: \"f93fb736-f6f2-4b03-9821-4036c36e2f72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:34:50.532325 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.532304 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f93fb736-f6f2-4b03-9821-4036c36e2f72-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tqq7s\" (UID: \"f93fb736-f6f2-4b03-9821-4036c36e2f72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:34:50.568509 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.568477 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:34:50.711988 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.711964 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s"] Aug 04 10:34:50.714412 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:34:50.714384 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf93fb736_f6f2_4b03_9821_4036c36e2f72.slice/crio-def62b70117768b3f2b85a472b89a19bb1eb522f882d57394fbe8d10695b6e20 WatchSource:0}: Error finding container def62b70117768b3f2b85a472b89a19bb1eb522f882d57394fbe8d10695b6e20: Status 404 returned error can't find the container with id def62b70117768b3f2b85a472b89a19bb1eb522f882d57394fbe8d10695b6e20 Aug 04 10:34:50.716507 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.716472 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:34:50.716623 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.716543 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:34:50.716623 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.716595 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:34:50.857536 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.857499 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" event={"ID":"f93fb736-f6f2-4b03-9821-4036c36e2f72","Type":"ContainerStarted","Data":"d81ca487a64c4e3b4a4fc8fc4b4a13260fbfa36eefa8db9c0c1baba8ea773215"} Aug 04 10:34:50.857718 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.857590 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" event={"ID":"f93fb736-f6f2-4b03-9821-4036c36e2f72","Type":"ContainerStarted","Data":"def62b70117768b3f2b85a472b89a19bb1eb522f882d57394fbe8d10695b6e20"} Aug 04 10:34:50.884828 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:50.884771 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" podStartSLOduration=0.884756279 podStartE2EDuration="884.756279ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:34:50.883700204 +0000 UTC m=+430.304587616" watchObservedRunningTime="2026-08-04 10:34:50.884756279 +0000 UTC m=+430.305643689" Aug 04 10:34:51.569228 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:51.569193 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:34:51.574256 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:51.574220 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:34:51.862508 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:51.862478 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:34:51.863451 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:34:51.863433 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:35:04.172793 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:35:04.172709 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:35:04.176027 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:35:04.176008 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj" Aug 04 10:35:04.179215 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:35:04.179195 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Aug 04 10:35:04.179731 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:35:04.179715 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-r7drb\"" Aug 04 10:35:04.198484 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:35:04.198456 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:35:04.229632 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:35:04.229600 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:35:04.238651 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:35:04.238617 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdkhl\" (UniqueName: \"kubernetes.io/projected/b3b6de8b-8543-42eb-9292-5a925d23fe0f-kube-api-access-gdkhl\") pod \"limitador-limitador-85d9db4d5d-jmqqj\" (UID: \"b3b6de8b-8543-42eb-9292-5a925d23fe0f\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj" Aug 04 10:35:04.238806 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:35:04.238703 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b3b6de8b-8543-42eb-9292-5a925d23fe0f-config-file\") pod \"limitador-limitador-85d9db4d5d-jmqqj\" (UID: \"b3b6de8b-8543-42eb-9292-5a925d23fe0f\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj" Aug 04 10:35:04.339830 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:35:04.339797 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdkhl\" (UniqueName: \"kubernetes.io/projected/b3b6de8b-8543-42eb-9292-5a925d23fe0f-kube-api-access-gdkhl\") pod \"limitador-limitador-85d9db4d5d-jmqqj\" (UID: \"b3b6de8b-8543-42eb-9292-5a925d23fe0f\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj" Aug 04 10:35:04.340001 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:35:04.339917 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b3b6de8b-8543-42eb-9292-5a925d23fe0f-config-file\") pod \"limitador-limitador-85d9db4d5d-jmqqj\" (UID: \"b3b6de8b-8543-42eb-9292-5a925d23fe0f\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj" Aug 04 10:35:04.340744 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:35:04.340722 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b3b6de8b-8543-42eb-9292-5a925d23fe0f-config-file\") pod \"limitador-limitador-85d9db4d5d-jmqqj\" (UID: \"b3b6de8b-8543-42eb-9292-5a925d23fe0f\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj" Aug 04 10:35:04.349020 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:35:04.348991 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdkhl\" (UniqueName: \"kubernetes.io/projected/b3b6de8b-8543-42eb-9292-5a925d23fe0f-kube-api-access-gdkhl\") pod \"limitador-limitador-85d9db4d5d-jmqqj\" (UID: \"b3b6de8b-8543-42eb-9292-5a925d23fe0f\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj" Aug 04 10:35:04.485843 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:35:04.485758 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj" Aug 04 10:35:04.630888 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:35:04.630849 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:35:04.634733 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:35:04.634704 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3b6de8b_8543_42eb_9292_5a925d23fe0f.slice/crio-3f29dacdd6b87dd0faece94aaa704915cba05f4fbae40e952ff9f99e7086eb41 WatchSource:0}: Error finding container 3f29dacdd6b87dd0faece94aaa704915cba05f4fbae40e952ff9f99e7086eb41: Status 404 returned error can't find the container with id 3f29dacdd6b87dd0faece94aaa704915cba05f4fbae40e952ff9f99e7086eb41 Aug 04 10:35:04.921009 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:35:04.920969 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj" event={"ID":"b3b6de8b-8543-42eb-9292-5a925d23fe0f","Type":"ContainerStarted","Data":"3f29dacdd6b87dd0faece94aaa704915cba05f4fbae40e952ff9f99e7086eb41"} Aug 04 10:35:06.931079 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:35:06.931043 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj" event={"ID":"b3b6de8b-8543-42eb-9292-5a925d23fe0f","Type":"ContainerStarted","Data":"4700c49f59c5c1a33678d8d76e87a4cffefb82df1a1e71e8922fed10032d55b3"} Aug 04 10:35:06.931510 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:35:06.931116 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj" Aug 04 10:35:06.951977 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:35:06.951926 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj" podStartSLOduration=1.70941443 podStartE2EDuration="2.951910353s" totalImagesPullingTime="1.242495923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:35:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:35:06.950987113 +0000 UTC m=+446.371874524" watchObservedRunningTime="2026-08-04 10:35:06.951910353 +0000 UTC m=+446.372797763" Aug 04 10:35:17.936682 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:35:17.936654 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj" Aug 04 10:36:20.055971 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:36:20.055925 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:36:29.297592 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:36:29.297472 2580 generic.go:356] "Generic (PLEG): container finished" podID="f93fb736-f6f2-4b03-9821-4036c36e2f72" containerID="d81ca487a64c4e3b4a4fc8fc4b4a13260fbfa36eefa8db9c0c1baba8ea773215" exitCode=0 Aug 04 10:36:29.297592 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:36:29.297540 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" event={"ID":"f93fb736-f6f2-4b03-9821-4036c36e2f72","Type":"ContainerDied","Data":"d81ca487a64c4e3b4a4fc8fc4b4a13260fbfa36eefa8db9c0c1baba8ea773215"} Aug 04 10:36:29.298037 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:36:29.297917 2580 scope.go:117] "RemoveContainer" containerID="d81ca487a64c4e3b4a4fc8fc4b4a13260fbfa36eefa8db9c0c1baba8ea773215" Aug 04 10:36:29.299694 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:36:29.299655 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:36:29.299905 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:36:29.299883 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:36:29.300034 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:36:29.300015 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:36:30.305073 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:36:30.305038 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" event={"ID":"f93fb736-f6f2-4b03-9821-4036c36e2f72","Type":"ContainerStarted","Data":"d1965cd09a5d3124405af23dd73397d336de34f0afa0b0b0e5dca7a1e05e979e"} Aug 04 10:36:30.569224 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:36:30.569126 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:36:30.569375 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:36:30.569337 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-tqq7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Aug 04 10:36:30.569425 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:36:30.569384 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" podUID="f93fb736-f6f2-4b03-9821-4036c36e2f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Aug 04 10:36:31.569013 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:36:31.568935 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-tqq7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Aug 04 10:36:31.569013 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:36:31.568994 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" podUID="f93fb736-f6f2-4b03-9821-4036c36e2f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Aug 04 10:36:32.568914 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:36:32.568876 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-tqq7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Aug 04 10:36:32.569083 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:36:32.568942 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" podUID="f93fb736-f6f2-4b03-9821-4036c36e2f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Aug 04 10:36:33.569804 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:36:33.569765 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-tqq7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Aug 04 10:36:33.570208 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:36:33.569827 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" podUID="f93fb736-f6f2-4b03-9821-4036c36e2f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Aug 04 10:36:34.569736 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:36:34.569693 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-tqq7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Aug 04 10:36:34.569891 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:36:34.569758 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" podUID="f93fb736-f6f2-4b03-9821-4036c36e2f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Aug 04 10:36:35.573059 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:36:35.573027 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:36:35.573473 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:36:35.573307 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:36:35.574223 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:36:35.574207 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tqq7s" Aug 04 10:36:42.360104 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:36:42.360064 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:36:47.024224 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:36:47.024182 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:37:09.053368 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:09.053326 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:37:13.233551 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:13.233509 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:37:40.547328 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:40.547292 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-tnqkq"] Aug 04 10:37:40.550706 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:40.550685 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tnqkq" Aug 04 10:37:40.553673 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:40.553648 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ps8jd\"" Aug 04 10:37:40.554982 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:40.554965 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Aug 04 10:37:40.555126 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:40.555050 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Aug 04 10:37:40.555126 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:40.555078 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Aug 04 10:37:40.561679 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:40.561659 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-tnqkq"] Aug 04 10:37:40.589523 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:40.589482 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b308251-1c5d-4a9a-adc3-a741a85c55f9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tnqkq\" (UID: \"5b308251-1c5d-4a9a-adc3-a741a85c55f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tnqkq" Aug 04 10:37:40.589715 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:40.589534 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kmx4\" (UniqueName: \"kubernetes.io/projected/5b308251-1c5d-4a9a-adc3-a741a85c55f9-kube-api-access-9kmx4\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tnqkq\" (UID: \"5b308251-1c5d-4a9a-adc3-a741a85c55f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tnqkq" Aug 04 10:37:40.589715 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:40.589638 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b308251-1c5d-4a9a-adc3-a741a85c55f9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tnqkq\" (UID: \"5b308251-1c5d-4a9a-adc3-a741a85c55f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tnqkq" Aug 04 10:37:40.589715 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:40.589682 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b308251-1c5d-4a9a-adc3-a741a85c55f9-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tnqkq\" (UID: \"5b308251-1c5d-4a9a-adc3-a741a85c55f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tnqkq" Aug 04 10:37:40.589715 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:40.589707 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b308251-1c5d-4a9a-adc3-a741a85c55f9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tnqkq\" (UID: \"5b308251-1c5d-4a9a-adc3-a741a85c55f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tnqkq" Aug 04 10:37:40.589856 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:40.589728 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b308251-1c5d-4a9a-adc3-a741a85c55f9-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tnqkq\" (UID: \"5b308251-1c5d-4a9a-adc3-a741a85c55f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tnqkq" Aug 04 10:37:40.691058 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:40.691021 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b308251-1c5d-4a9a-adc3-a741a85c55f9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tnqkq\" (UID: \"5b308251-1c5d-4a9a-adc3-a741a85c55f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tnqkq" Aug 04 10:37:40.691058 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:40.691062 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b308251-1c5d-4a9a-adc3-a741a85c55f9-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tnqkq\" (UID: \"5b308251-1c5d-4a9a-adc3-a741a85c55f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tnqkq" Aug 04 10:37:40.691319 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:40.691095 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b308251-1c5d-4a9a-adc3-a741a85c55f9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tnqkq\" (UID: \"5b308251-1c5d-4a9a-adc3-a741a85c55f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tnqkq" Aug 04 10:37:40.691319 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:40.691122 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kmx4\" (UniqueName: \"kubernetes.io/projected/5b308251-1c5d-4a9a-adc3-a741a85c55f9-kube-api-access-9kmx4\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tnqkq\" (UID: \"5b308251-1c5d-4a9a-adc3-a741a85c55f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tnqkq" Aug 04 10:37:40.691319 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:40.691157 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b308251-1c5d-4a9a-adc3-a741a85c55f9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tnqkq\" (UID: \"5b308251-1c5d-4a9a-adc3-a741a85c55f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tnqkq" Aug 04 10:37:40.691319 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:40.691187 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b308251-1c5d-4a9a-adc3-a741a85c55f9-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tnqkq\" (UID: \"5b308251-1c5d-4a9a-adc3-a741a85c55f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tnqkq" Aug 04 10:37:40.691645 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:40.691621 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b308251-1c5d-4a9a-adc3-a741a85c55f9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tnqkq\" (UID: \"5b308251-1c5d-4a9a-adc3-a741a85c55f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tnqkq" Aug 04 10:37:40.691735 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:40.691642 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b308251-1c5d-4a9a-adc3-a741a85c55f9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tnqkq\" (UID: \"5b308251-1c5d-4a9a-adc3-a741a85c55f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tnqkq" Aug 04 10:37:40.691735 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:40.691671 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b308251-1c5d-4a9a-adc3-a741a85c55f9-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tnqkq\" (UID: \"5b308251-1c5d-4a9a-adc3-a741a85c55f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tnqkq" Aug 04 10:37:40.693535 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:40.693511 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b308251-1c5d-4a9a-adc3-a741a85c55f9-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tnqkq\" (UID: \"5b308251-1c5d-4a9a-adc3-a741a85c55f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tnqkq" Aug 04 10:37:40.693790 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:40.693777 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b308251-1c5d-4a9a-adc3-a741a85c55f9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tnqkq\" (UID: \"5b308251-1c5d-4a9a-adc3-a741a85c55f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tnqkq" Aug 04 10:37:40.701664 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:40.701646 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kmx4\" (UniqueName: \"kubernetes.io/projected/5b308251-1c5d-4a9a-adc3-a741a85c55f9-kube-api-access-9kmx4\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tnqkq\" (UID: \"5b308251-1c5d-4a9a-adc3-a741a85c55f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tnqkq" Aug 04 10:37:40.861615 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:40.861551 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tnqkq" Aug 04 10:37:40.995415 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:40.995392 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tb4zk_c5922147-cb42-4f62-9880-8deaa51a4b0f/ovn-acl-logging/0.log" Aug 04 10:37:40.996091 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:40.996072 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tb4zk_c5922147-cb42-4f62-9880-8deaa51a4b0f/ovn-acl-logging/0.log" Aug 04 10:37:41.003226 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:41.003206 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-tnqkq"] Aug 04 10:37:41.005048 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:37:41.005026 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b308251_1c5d_4a9a_adc3_a741a85c55f9.slice/crio-bd7d6cf870e3f25ed20e9f08cc01f22f9e7426cee911897f6ffac0cf52f4421b WatchSource:0}: Error finding container bd7d6cf870e3f25ed20e9f08cc01f22f9e7426cee911897f6ffac0cf52f4421b: Status 404 returned error can't find the container with id bd7d6cf870e3f25ed20e9f08cc01f22f9e7426cee911897f6ffac0cf52f4421b Aug 04 10:37:41.627072 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:41.627028 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tnqkq" event={"ID":"5b308251-1c5d-4a9a-adc3-a741a85c55f9","Type":"ContainerStarted","Data":"bd7d6cf870e3f25ed20e9f08cc01f22f9e7426cee911897f6ffac0cf52f4421b"} Aug 04 10:37:42.633657 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:42.633609 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tnqkq" event={"ID":"5b308251-1c5d-4a9a-adc3-a741a85c55f9","Type":"ContainerStarted","Data":"47d3be65bc412dffd50c75043c90f3e43323aa60ac7a7760076c507421ef1bd1"} Aug 04 10:37:42.655005 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:42.654955 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tnqkq" podStartSLOduration=1.57656603 podStartE2EDuration="2.654938273s" totalImagesPullingTime="1.078372243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:37:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:37:42.654111893 +0000 UTC m=+602.074999304" watchObservedRunningTime="2026-08-04 10:37:42.654938273 +0000 UTC m=+602.075825684" Aug 04 10:37:42.850831 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:42.850795 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:37:49.831602 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:49.831546 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:37:50.862544 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:50.862495 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tnqkq" Aug 04 10:37:50.862984 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:50.862601 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tnqkq" Aug 04 10:37:50.875493 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:50.875449 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tnqkq" Aug 04 10:37:51.687700 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:37:51.687671 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tnqkq" Aug 04 10:39:25.141339 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:39:25.141301 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:39:35.432135 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:39:35.432100 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:39:47.140596 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:39:47.140548 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:41:25.541193 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:41:25.541155 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:41:35.828973 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:41:35.828932 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:42:34.536941 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:42:34.536906 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:42:41.020398 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:42:41.020360 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tb4zk_c5922147-cb42-4f62-9880-8deaa51a4b0f/ovn-acl-logging/0.log" Aug 04 10:42:41.021319 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:42:41.021299 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tb4zk_c5922147-cb42-4f62-9880-8deaa51a4b0f/ovn-acl-logging/0.log" Aug 04 10:42:41.436811 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:42:41.436769 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:42:56.835294 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:42:56.835256 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:43:02.743201 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:43:02.743162 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:43:12.245645 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:43:12.245612 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:43:18.838029 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:43:18.837992 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:43:32.927802 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:43:32.927765 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:43:38.037250 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:43:38.037214 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:43:43.333167 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:43:43.333132 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:44:05.138553 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:44:05.138473 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:44:15.827133 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:44:15.827098 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:44:38.933902 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:44:38.930319 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:44:49.334490 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:44:49.334453 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:44:54.629504 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:44:54.629465 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:45:01.428428 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:45:01.428391 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:45:09.435299 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:45:09.435265 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:45:15.033837 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:45:15.033784 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:45:21.436595 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:45:21.436545 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:45:27.926367 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:45:27.926329 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:45:41.836142 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:45:41.836099 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:45:51.035794 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:45:51.035754 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:46:00.626084 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:46:00.626046 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:46:05.830468 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:46:05.830429 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:46:13.728034 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:46:13.728000 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:46:20.742074 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:46:20.742037 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:46:26.035776 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:46:26.035740 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:46:36.540031 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:46:36.539992 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:47:32.084291 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:47:32.084206 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-4drcn"] Aug 04 10:47:32.084835 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:47:32.084395 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-4drcn" podUID="af82c5b5-69ad-4007-84bf-fb7380b9577e" containerName="manager" containerID="cri-o://b8d97c3c5ff4dc1677d9889e6a62d22af769574f97e1eb2e623e7da3478c9bfa" gracePeriod=10 Aug 04 10:47:32.335010 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:47:32.334940 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-4drcn" Aug 04 10:47:32.340000 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:47:32.339965 2580 generic.go:356] "Generic (PLEG): container finished" podID="af82c5b5-69ad-4007-84bf-fb7380b9577e" containerID="b8d97c3c5ff4dc1677d9889e6a62d22af769574f97e1eb2e623e7da3478c9bfa" exitCode=0 Aug 04 10:47:32.340148 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:47:32.340008 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-4drcn" Aug 04 10:47:32.340148 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:47:32.340050 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-4drcn" event={"ID":"af82c5b5-69ad-4007-84bf-fb7380b9577e","Type":"ContainerDied","Data":"b8d97c3c5ff4dc1677d9889e6a62d22af769574f97e1eb2e623e7da3478c9bfa"} Aug 04 10:47:32.340148 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:47:32.340119 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-4drcn" event={"ID":"af82c5b5-69ad-4007-84bf-fb7380b9577e","Type":"ContainerDied","Data":"0302e598e7b3080f92a5ac60937f1d3635df03bd38964c05c8f6cfa37442c2f8"} Aug 04 10:47:32.340303 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:47:32.340149 2580 scope.go:117] "RemoveContainer" containerID="b8d97c3c5ff4dc1677d9889e6a62d22af769574f97e1eb2e623e7da3478c9bfa" Aug 04 10:47:32.350339 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:47:32.350321 2580 scope.go:117] "RemoveContainer" containerID="b8d97c3c5ff4dc1677d9889e6a62d22af769574f97e1eb2e623e7da3478c9bfa" Aug 04 10:47:32.350646 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:47:32.350623 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8d97c3c5ff4dc1677d9889e6a62d22af769574f97e1eb2e623e7da3478c9bfa\": container with ID starting with b8d97c3c5ff4dc1677d9889e6a62d22af769574f97e1eb2e623e7da3478c9bfa not found: ID does not exist" containerID="b8d97c3c5ff4dc1677d9889e6a62d22af769574f97e1eb2e623e7da3478c9bfa" Aug 04 10:47:32.350752 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:47:32.350654 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8d97c3c5ff4dc1677d9889e6a62d22af769574f97e1eb2e623e7da3478c9bfa"} err="failed to get container status \"b8d97c3c5ff4dc1677d9889e6a62d22af769574f97e1eb2e623e7da3478c9bfa\": rpc error: code = NotFound desc = could not find container \"b8d97c3c5ff4dc1677d9889e6a62d22af769574f97e1eb2e623e7da3478c9bfa\": container with ID starting with b8d97c3c5ff4dc1677d9889e6a62d22af769574f97e1eb2e623e7da3478c9bfa not found: ID does not exist" Aug 04 10:47:32.411701 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:47:32.411655 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/af82c5b5-69ad-4007-84bf-fb7380b9577e-extensions-socket-volume\") pod \"af82c5b5-69ad-4007-84bf-fb7380b9577e\" (UID: \"af82c5b5-69ad-4007-84bf-fb7380b9577e\") " Aug 04 10:47:32.411880 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:47:32.411737 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvlz2\" (UniqueName: \"kubernetes.io/projected/af82c5b5-69ad-4007-84bf-fb7380b9577e-kube-api-access-nvlz2\") pod \"af82c5b5-69ad-4007-84bf-fb7380b9577e\" (UID: \"af82c5b5-69ad-4007-84bf-fb7380b9577e\") " Aug 04 10:47:32.412001 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:47:32.411977 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af82c5b5-69ad-4007-84bf-fb7380b9577e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "af82c5b5-69ad-4007-84bf-fb7380b9577e" (UID: "af82c5b5-69ad-4007-84bf-fb7380b9577e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:47:32.414154 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:47:32.414107 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af82c5b5-69ad-4007-84bf-fb7380b9577e-kube-api-access-nvlz2" (OuterVolumeSpecName: "kube-api-access-nvlz2") pod "af82c5b5-69ad-4007-84bf-fb7380b9577e" (UID: "af82c5b5-69ad-4007-84bf-fb7380b9577e"). InnerVolumeSpecName "kube-api-access-nvlz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:47:32.512986 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:47:32.512943 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/af82c5b5-69ad-4007-84bf-fb7380b9577e-extensions-socket-volume\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:47:32.512986 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:47:32.512979 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvlz2\" (UniqueName: \"kubernetes.io/projected/af82c5b5-69ad-4007-84bf-fb7380b9577e-kube-api-access-nvlz2\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:47:32.669845 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:47:32.669813 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-4drcn"] Aug 04 10:47:32.675212 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:47:32.675183 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-4drcn"] Aug 04 10:47:33.099227 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:47:33.099146 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af82c5b5-69ad-4007-84bf-fb7380b9577e" path="/var/lib/kubelet/pods/af82c5b5-69ad-4007-84bf-fb7380b9577e/volumes" Aug 04 10:47:41.045928 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:47:41.045893 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tb4zk_c5922147-cb42-4f62-9880-8deaa51a4b0f/ovn-acl-logging/0.log" Aug 04 10:47:41.049406 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:47:41.049383 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tb4zk_c5922147-cb42-4f62-9880-8deaa51a4b0f/ovn-acl-logging/0.log" Aug 04 10:48:32.816835 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:48:32.816791 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9t69l"] Aug 04 10:48:32.817340 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:48:32.817153 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af82c5b5-69ad-4007-84bf-fb7380b9577e" containerName="manager" Aug 04 10:48:32.817340 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:48:32.817165 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="af82c5b5-69ad-4007-84bf-fb7380b9577e" containerName="manager" Aug 04 10:48:32.817340 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:48:32.817226 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="af82c5b5-69ad-4007-84bf-fb7380b9577e" containerName="manager" Aug 04 10:48:32.820473 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:48:32.820453 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9t69l" Aug 04 10:48:32.823869 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:48:32.823841 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-spfp7\"" Aug 04 10:48:32.834591 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:48:32.834532 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9t69l"] Aug 04 10:48:32.936959 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:48:32.936883 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlpxt\" (UniqueName: \"kubernetes.io/projected/690f2b91-2762-4b2e-ac6c-561d645c73a6-kube-api-access-mlpxt\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-9t69l\" (UID: \"690f2b91-2762-4b2e-ac6c-561d645c73a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9t69l" Aug 04 10:48:32.937137 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:48:32.936977 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/690f2b91-2762-4b2e-ac6c-561d645c73a6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-9t69l\" (UID: \"690f2b91-2762-4b2e-ac6c-561d645c73a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9t69l" Aug 04 10:48:33.038118 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:48:33.038074 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlpxt\" (UniqueName: \"kubernetes.io/projected/690f2b91-2762-4b2e-ac6c-561d645c73a6-kube-api-access-mlpxt\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-9t69l\" (UID: \"690f2b91-2762-4b2e-ac6c-561d645c73a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9t69l" Aug 04 10:48:33.038292 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:48:33.038127 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/690f2b91-2762-4b2e-ac6c-561d645c73a6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-9t69l\" (UID: \"690f2b91-2762-4b2e-ac6c-561d645c73a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9t69l" Aug 04 10:48:33.038481 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:48:33.038463 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/690f2b91-2762-4b2e-ac6c-561d645c73a6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-9t69l\" (UID: \"690f2b91-2762-4b2e-ac6c-561d645c73a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9t69l" Aug 04 10:48:33.047862 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:48:33.047830 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlpxt\" (UniqueName: \"kubernetes.io/projected/690f2b91-2762-4b2e-ac6c-561d645c73a6-kube-api-access-mlpxt\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-9t69l\" (UID: \"690f2b91-2762-4b2e-ac6c-561d645c73a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9t69l" Aug 04 10:48:33.131300 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:48:33.131269 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9t69l" Aug 04 10:48:33.283196 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:48:33.283171 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9t69l"] Aug 04 10:48:33.285319 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:48:33.285291 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod690f2b91_2762_4b2e_ac6c_561d645c73a6.slice/crio-09458021610fc24a0c2aca47dda79bd017c9f2bab4be71cd2293d8cc49bb87bd WatchSource:0}: Error finding container 09458021610fc24a0c2aca47dda79bd017c9f2bab4be71cd2293d8cc49bb87bd: Status 404 returned error can't find the container with id 09458021610fc24a0c2aca47dda79bd017c9f2bab4be71cd2293d8cc49bb87bd Aug 04 10:48:33.612005 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:48:33.611970 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9t69l" event={"ID":"690f2b91-2762-4b2e-ac6c-561d645c73a6","Type":"ContainerStarted","Data":"632c0b35701ca7fa1d3f5497b3efa6f374fa90dbfe49cc4384ea80e52c26b937"} Aug 04 10:48:33.612178 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:48:33.612021 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9t69l" event={"ID":"690f2b91-2762-4b2e-ac6c-561d645c73a6","Type":"ContainerStarted","Data":"09458021610fc24a0c2aca47dda79bd017c9f2bab4be71cd2293d8cc49bb87bd"} Aug 04 10:48:33.612178 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:48:33.612067 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9t69l" Aug 04 10:48:33.639578 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:48:33.639501 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9t69l" podStartSLOduration=1.639483082 podStartE2EDuration="1.639483082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:48:33.638106065 +0000 UTC m=+1253.058993476" watchObservedRunningTime="2026-08-04 10:48:33.639483082 +0000 UTC m=+1253.060370470" Aug 04 10:48:44.619523 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:48:44.619494 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9t69l" Aug 04 10:48:54.635323 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:48:54.635276 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:49:02.131633 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:49:02.131596 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:49:22.843555 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:49:22.843518 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:49:33.840894 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:49:33.840863 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:49:40.333930 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:49:40.333890 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:49:50.242163 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:49:50.242129 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:49:55.240417 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:49:55.240383 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:50:05.738994 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:50:05.738903 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:50:11.531022 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:50:11.530983 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:50:21.128092 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:50:21.128054 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:50:26.740419 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:50:26.740377 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:50:36.341502 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:50:36.341466 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:50:42.748713 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:50:42.748673 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:50:52.536503 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:50:52.536463 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:51:16.862495 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:16.862460 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:51:22.038042 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:22.038003 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:51:35.550155 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:35.550117 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg"] Aug 04 10:51:35.553546 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:35.553531 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg" Aug 04 10:51:35.556708 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:35.556687 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-f9925d20-kserve-self-signed-certs\"" Aug 04 10:51:35.573179 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:35.573152 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg"] Aug 04 10:51:35.631113 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:35.631076 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6lrr\" (UniqueName: \"kubernetes.io/projected/2904dd2c-68f6-4e6a-9cf6-5063346046b8-kube-api-access-k6lrr\") pod \"e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg\" (UID: \"2904dd2c-68f6-4e6a-9cf6-5063346046b8\") " pod="llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg" Aug 04 10:51:35.631296 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:35.631123 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2904dd2c-68f6-4e6a-9cf6-5063346046b8-tmp-dir\") pod \"e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg\" (UID: \"2904dd2c-68f6-4e6a-9cf6-5063346046b8\") " pod="llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg" Aug 04 10:51:35.631296 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:35.631157 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2904dd2c-68f6-4e6a-9cf6-5063346046b8-model-cache\") pod \"e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg\" (UID: \"2904dd2c-68f6-4e6a-9cf6-5063346046b8\") " pod="llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg" Aug 04 10:51:35.631296 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:35.631252 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2904dd2c-68f6-4e6a-9cf6-5063346046b8-home\") pod \"e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg\" (UID: \"2904dd2c-68f6-4e6a-9cf6-5063346046b8\") " pod="llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg" Aug 04 10:51:35.631296 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:35.631283 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2904dd2c-68f6-4e6a-9cf6-5063346046b8-dshm\") pod \"e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg\" (UID: \"2904dd2c-68f6-4e6a-9cf6-5063346046b8\") " pod="llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg" Aug 04 10:51:35.631445 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:35.631309 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2904dd2c-68f6-4e6a-9cf6-5063346046b8-tls-certs\") pod \"e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg\" (UID: \"2904dd2c-68f6-4e6a-9cf6-5063346046b8\") " pod="llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg" Aug 04 10:51:35.731970 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:35.731936 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2904dd2c-68f6-4e6a-9cf6-5063346046b8-tmp-dir\") pod \"e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg\" (UID: \"2904dd2c-68f6-4e6a-9cf6-5063346046b8\") " pod="llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg" Aug 04 10:51:35.732157 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:35.731984 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2904dd2c-68f6-4e6a-9cf6-5063346046b8-model-cache\") pod \"e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg\" (UID: \"2904dd2c-68f6-4e6a-9cf6-5063346046b8\") " pod="llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg" Aug 04 10:51:35.732157 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:35.732053 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2904dd2c-68f6-4e6a-9cf6-5063346046b8-home\") pod \"e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg\" (UID: \"2904dd2c-68f6-4e6a-9cf6-5063346046b8\") " pod="llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg" Aug 04 10:51:35.732157 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:35.732076 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2904dd2c-68f6-4e6a-9cf6-5063346046b8-dshm\") pod \"e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg\" (UID: \"2904dd2c-68f6-4e6a-9cf6-5063346046b8\") " pod="llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg" Aug 04 10:51:35.732157 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:35.732108 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2904dd2c-68f6-4e6a-9cf6-5063346046b8-tls-certs\") pod \"e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg\" (UID: \"2904dd2c-68f6-4e6a-9cf6-5063346046b8\") " pod="llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg" Aug 04 10:51:35.732157 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:35.732155 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6lrr\" (UniqueName: \"kubernetes.io/projected/2904dd2c-68f6-4e6a-9cf6-5063346046b8-kube-api-access-k6lrr\") pod \"e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg\" (UID: \"2904dd2c-68f6-4e6a-9cf6-5063346046b8\") " pod="llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg" Aug 04 10:51:35.732398 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:35.732375 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2904dd2c-68f6-4e6a-9cf6-5063346046b8-tmp-dir\") pod \"e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg\" (UID: \"2904dd2c-68f6-4e6a-9cf6-5063346046b8\") " pod="llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg" Aug 04 10:51:35.732457 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:35.732437 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2904dd2c-68f6-4e6a-9cf6-5063346046b8-model-cache\") pod \"e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg\" (UID: \"2904dd2c-68f6-4e6a-9cf6-5063346046b8\") " pod="llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg" Aug 04 10:51:35.732506 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:35.732444 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2904dd2c-68f6-4e6a-9cf6-5063346046b8-home\") pod \"e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg\" (UID: \"2904dd2c-68f6-4e6a-9cf6-5063346046b8\") " pod="llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg" Aug 04 10:51:35.734623 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:35.734601 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2904dd2c-68f6-4e6a-9cf6-5063346046b8-dshm\") pod \"e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg\" (UID: \"2904dd2c-68f6-4e6a-9cf6-5063346046b8\") " pod="llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg" Aug 04 10:51:35.734798 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:35.734779 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2904dd2c-68f6-4e6a-9cf6-5063346046b8-tls-certs\") pod \"e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg\" (UID: \"2904dd2c-68f6-4e6a-9cf6-5063346046b8\") " pod="llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg" Aug 04 10:51:35.748763 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:35.748738 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6lrr\" (UniqueName: \"kubernetes.io/projected/2904dd2c-68f6-4e6a-9cf6-5063346046b8-kube-api-access-k6lrr\") pod \"e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg\" (UID: \"2904dd2c-68f6-4e6a-9cf6-5063346046b8\") " pod="llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg" Aug 04 10:51:35.861965 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:35.861928 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg" Aug 04 10:51:36.031277 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:36.031241 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg"] Aug 04 10:51:36.034358 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:51:36.034332 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2904dd2c_68f6_4e6a_9cf6_5063346046b8.slice/crio-2d1f584609d84668de91a2fef97ef541058490be5748fcd4943768124a98ab35 WatchSource:0}: Error finding container 2d1f584609d84668de91a2fef97ef541058490be5748fcd4943768124a98ab35: Status 404 returned error can't find the container with id 2d1f584609d84668de91a2fef97ef541058490be5748fcd4943768124a98ab35 Aug 04 10:51:36.445111 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:36.445077 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg" event={"ID":"2904dd2c-68f6-4e6a-9cf6-5063346046b8","Type":"ContainerStarted","Data":"fa0c7b30f404b625af5fe9acd398ec0c48a69ab52a79113d2c8d7d535547c5bd"} Aug 04 10:51:36.445111 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:36.445116 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg" event={"ID":"2904dd2c-68f6-4e6a-9cf6-5063346046b8","Type":"ContainerStarted","Data":"2d1f584609d84668de91a2fef97ef541058490be5748fcd4943768124a98ab35"} Aug 04 10:51:36.464835 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:36.464781 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg" podStartSLOduration=1.464761269 podStartE2EDuration="1.464761269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:51:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:51:36.463519798 +0000 UTC m=+1435.884407208" watchObservedRunningTime="2026-08-04 10:51:36.464761269 +0000 UTC m=+1435.885648679" Aug 04 10:51:38.171267 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:38.171235 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:51:42.803850 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:42.803807 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2"] Aug 04 10:51:42.807278 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:42.807254 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2" Aug 04 10:51:42.810386 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:42.810361 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-f9925d20-kserve-self-signed-certs\"" Aug 04 10:51:42.821941 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:42.821913 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2"] Aug 04 10:51:42.894780 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:42.894743 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75a22383-40e4-4c42-9f05-036bc4db4576-dshm\") pod \"e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2\" (UID: \"75a22383-40e4-4c42-9f05-036bc4db4576\") " pod="llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2" Aug 04 10:51:42.894780 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:42.894781 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75a22383-40e4-4c42-9f05-036bc4db4576-tmp-dir\") pod \"e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2\" (UID: \"75a22383-40e4-4c42-9f05-036bc4db4576\") " pod="llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2" Aug 04 10:51:42.895005 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:42.894812 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkkmz\" (UniqueName: \"kubernetes.io/projected/75a22383-40e4-4c42-9f05-036bc4db4576-kube-api-access-rkkmz\") pod \"e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2\" (UID: \"75a22383-40e4-4c42-9f05-036bc4db4576\") " pod="llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2" Aug 04 10:51:42.895005 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:42.894903 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75a22383-40e4-4c42-9f05-036bc4db4576-model-cache\") pod \"e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2\" (UID: \"75a22383-40e4-4c42-9f05-036bc4db4576\") " pod="llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2" Aug 04 10:51:42.895005 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:42.894968 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75a22383-40e4-4c42-9f05-036bc4db4576-home\") pod \"e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2\" (UID: \"75a22383-40e4-4c42-9f05-036bc4db4576\") " pod="llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2" Aug 04 10:51:42.895131 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:42.895029 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a22383-40e4-4c42-9f05-036bc4db4576-tls-certs\") pod \"e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2\" (UID: \"75a22383-40e4-4c42-9f05-036bc4db4576\") " pod="llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2" Aug 04 10:51:42.995956 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:42.995915 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75a22383-40e4-4c42-9f05-036bc4db4576-model-cache\") pod \"e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2\" (UID: \"75a22383-40e4-4c42-9f05-036bc4db4576\") " pod="llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2" Aug 04 10:51:42.996177 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:42.995986 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75a22383-40e4-4c42-9f05-036bc4db4576-home\") pod \"e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2\" (UID: \"75a22383-40e4-4c42-9f05-036bc4db4576\") " pod="llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2" Aug 04 10:51:42.996177 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:42.996019 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a22383-40e4-4c42-9f05-036bc4db4576-tls-certs\") pod \"e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2\" (UID: \"75a22383-40e4-4c42-9f05-036bc4db4576\") " pod="llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2" Aug 04 10:51:42.996177 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:42.996040 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75a22383-40e4-4c42-9f05-036bc4db4576-dshm\") pod \"e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2\" (UID: \"75a22383-40e4-4c42-9f05-036bc4db4576\") " pod="llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2" Aug 04 10:51:42.996177 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:42.996059 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75a22383-40e4-4c42-9f05-036bc4db4576-tmp-dir\") pod \"e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2\" (UID: \"75a22383-40e4-4c42-9f05-036bc4db4576\") " pod="llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2" Aug 04 10:51:42.996177 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:42.996103 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkkmz\" (UniqueName: \"kubernetes.io/projected/75a22383-40e4-4c42-9f05-036bc4db4576-kube-api-access-rkkmz\") pod \"e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2\" (UID: \"75a22383-40e4-4c42-9f05-036bc4db4576\") " pod="llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2" Aug 04 10:51:42.996402 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:42.996380 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75a22383-40e4-4c42-9f05-036bc4db4576-model-cache\") pod \"e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2\" (UID: \"75a22383-40e4-4c42-9f05-036bc4db4576\") " pod="llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2" Aug 04 10:51:42.996444 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:42.996417 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75a22383-40e4-4c42-9f05-036bc4db4576-home\") pod \"e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2\" (UID: \"75a22383-40e4-4c42-9f05-036bc4db4576\") " pod="llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2" Aug 04 10:51:42.996482 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:42.996466 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75a22383-40e4-4c42-9f05-036bc4db4576-tmp-dir\") pod \"e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2\" (UID: \"75a22383-40e4-4c42-9f05-036bc4db4576\") " pod="llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2" Aug 04 10:51:42.998537 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:42.998518 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75a22383-40e4-4c42-9f05-036bc4db4576-dshm\") pod \"e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2\" (UID: \"75a22383-40e4-4c42-9f05-036bc4db4576\") " pod="llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2" Aug 04 10:51:42.998835 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:42.998817 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a22383-40e4-4c42-9f05-036bc4db4576-tls-certs\") pod \"e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2\" (UID: \"75a22383-40e4-4c42-9f05-036bc4db4576\") " pod="llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2" Aug 04 10:51:43.006160 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:43.006135 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkkmz\" (UniqueName: \"kubernetes.io/projected/75a22383-40e4-4c42-9f05-036bc4db4576-kube-api-access-rkkmz\") pod \"e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2\" (UID: \"75a22383-40e4-4c42-9f05-036bc4db4576\") " pod="llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2" Aug 04 10:51:43.116770 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:43.116738 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2" Aug 04 10:51:43.260544 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:43.260372 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2"] Aug 04 10:51:43.263256 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:51:43.263224 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75a22383_40e4_4c42_9f05_036bc4db4576.slice/crio-63e81617d9db14056bb6cfde2d2967abe48eece444c532915fd9e2262dddc434 WatchSource:0}: Error finding container 63e81617d9db14056bb6cfde2d2967abe48eece444c532915fd9e2262dddc434: Status 404 returned error can't find the container with id 63e81617d9db14056bb6cfde2d2967abe48eece444c532915fd9e2262dddc434 Aug 04 10:51:43.479220 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:43.479122 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2" event={"ID":"75a22383-40e4-4c42-9f05-036bc4db4576","Type":"ContainerStarted","Data":"f312e4edc08bf81c823d545c3c52465f3fe4781be72e4e7be0762a18ddaceecf"} Aug 04 10:51:43.479220 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:43.479171 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2" event={"ID":"75a22383-40e4-4c42-9f05-036bc4db4576","Type":"ContainerStarted","Data":"63e81617d9db14056bb6cfde2d2967abe48eece444c532915fd9e2262dddc434"} Aug 04 10:51:43.500409 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:43.500354 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2" podStartSLOduration=1.500339085 podStartE2EDuration="1.500339085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:51:43.499242504 +0000 UTC m=+1442.920129936" watchObservedRunningTime="2026-08-04 10:51:43.500339085 +0000 UTC m=+1442.921226496" Aug 04 10:51:45.862708 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:45.862667 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg" Aug 04 10:51:45.863199 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:45.862757 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg" Aug 04 10:51:45.875245 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:45.875218 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg" Aug 04 10:51:46.504266 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:46.504236 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg" Aug 04 10:51:51.062862 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:51.062831 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:51:53.117488 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:53.117459 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2" Aug 04 10:51:53.117919 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:53.117510 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2" Aug 04 10:51:53.129925 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:53.129895 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2" Aug 04 10:51:53.535190 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:51:53.535106 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2" Aug 04 10:52:36.249474 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:36.249441 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg"] Aug 04 10:52:37.236671 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:37.236640 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:52:37.839096 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:37.839062 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2"] Aug 04 10:52:41.071227 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:41.071203 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tb4zk_c5922147-cb42-4f62-9880-8deaa51a4b0f/ovn-acl-logging/0.log" Aug 04 10:52:41.076691 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:41.076669 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tb4zk_c5922147-cb42-4f62-9880-8deaa51a4b0f/ovn-acl-logging/0.log" Aug 04 10:52:44.971216 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:44.971177 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:52:51.268933 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:51.268872 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg" podUID="2904dd2c-68f6-4e6a-9cf6-5063346046b8" containerName="main" containerID="cri-o://fa0c7b30f404b625af5fe9acd398ec0c48a69ab52a79113d2c8d7d535547c5bd" gracePeriod=45 Aug 04 10:52:51.528216 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:51.528159 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg" Aug 04 10:52:51.620268 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:51.620232 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2904dd2c-68f6-4e6a-9cf6-5063346046b8-model-cache\") pod \"2904dd2c-68f6-4e6a-9cf6-5063346046b8\" (UID: \"2904dd2c-68f6-4e6a-9cf6-5063346046b8\") " Aug 04 10:52:51.620445 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:51.620291 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2904dd2c-68f6-4e6a-9cf6-5063346046b8-tls-certs\") pod \"2904dd2c-68f6-4e6a-9cf6-5063346046b8\" (UID: \"2904dd2c-68f6-4e6a-9cf6-5063346046b8\") " Aug 04 10:52:51.620445 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:51.620319 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2904dd2c-68f6-4e6a-9cf6-5063346046b8-dshm\") pod \"2904dd2c-68f6-4e6a-9cf6-5063346046b8\" (UID: \"2904dd2c-68f6-4e6a-9cf6-5063346046b8\") " Aug 04 10:52:51.620445 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:51.620354 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2904dd2c-68f6-4e6a-9cf6-5063346046b8-tmp-dir\") pod \"2904dd2c-68f6-4e6a-9cf6-5063346046b8\" (UID: \"2904dd2c-68f6-4e6a-9cf6-5063346046b8\") " Aug 04 10:52:51.620445 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:51.620413 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6lrr\" (UniqueName: \"kubernetes.io/projected/2904dd2c-68f6-4e6a-9cf6-5063346046b8-kube-api-access-k6lrr\") pod \"2904dd2c-68f6-4e6a-9cf6-5063346046b8\" (UID: \"2904dd2c-68f6-4e6a-9cf6-5063346046b8\") " Aug 04 10:52:51.620445 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:51.620438 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2904dd2c-68f6-4e6a-9cf6-5063346046b8-home\") pod \"2904dd2c-68f6-4e6a-9cf6-5063346046b8\" (UID: \"2904dd2c-68f6-4e6a-9cf6-5063346046b8\") " Aug 04 10:52:51.620729 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:51.620508 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2904dd2c-68f6-4e6a-9cf6-5063346046b8-model-cache" (OuterVolumeSpecName: "model-cache") pod "2904dd2c-68f6-4e6a-9cf6-5063346046b8" (UID: "2904dd2c-68f6-4e6a-9cf6-5063346046b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:52:51.620729 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:51.620665 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2904dd2c-68f6-4e6a-9cf6-5063346046b8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2904dd2c-68f6-4e6a-9cf6-5063346046b8" (UID: "2904dd2c-68f6-4e6a-9cf6-5063346046b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:52:51.620729 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:51.620724 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2904dd2c-68f6-4e6a-9cf6-5063346046b8-home" (OuterVolumeSpecName: "home") pod "2904dd2c-68f6-4e6a-9cf6-5063346046b8" (UID: "2904dd2c-68f6-4e6a-9cf6-5063346046b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:52:51.620847 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:51.620735 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2904dd2c-68f6-4e6a-9cf6-5063346046b8-model-cache\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:52:51.622777 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:51.622756 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2904dd2c-68f6-4e6a-9cf6-5063346046b8-dshm" (OuterVolumeSpecName: "dshm") pod "2904dd2c-68f6-4e6a-9cf6-5063346046b8" (UID: "2904dd2c-68f6-4e6a-9cf6-5063346046b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:52:51.622777 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:51.622769 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2904dd2c-68f6-4e6a-9cf6-5063346046b8-kube-api-access-k6lrr" (OuterVolumeSpecName: "kube-api-access-k6lrr") pod "2904dd2c-68f6-4e6a-9cf6-5063346046b8" (UID: "2904dd2c-68f6-4e6a-9cf6-5063346046b8"). InnerVolumeSpecName "kube-api-access-k6lrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:52:51.622951 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:51.622800 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2904dd2c-68f6-4e6a-9cf6-5063346046b8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2904dd2c-68f6-4e6a-9cf6-5063346046b8" (UID: "2904dd2c-68f6-4e6a-9cf6-5063346046b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 10:52:51.721220 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:51.721180 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2904dd2c-68f6-4e6a-9cf6-5063346046b8-tls-certs\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:52:51.721220 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:51.721208 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2904dd2c-68f6-4e6a-9cf6-5063346046b8-dshm\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:52:51.721220 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:51.721217 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2904dd2c-68f6-4e6a-9cf6-5063346046b8-tmp-dir\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:52:51.721220 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:51.721226 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6lrr\" (UniqueName: \"kubernetes.io/projected/2904dd2c-68f6-4e6a-9cf6-5063346046b8-kube-api-access-k6lrr\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:52:51.721486 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:51.721236 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2904dd2c-68f6-4e6a-9cf6-5063346046b8-home\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:52:51.791042 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:51.790953 2580 generic.go:356] "Generic (PLEG): container finished" podID="2904dd2c-68f6-4e6a-9cf6-5063346046b8" containerID="fa0c7b30f404b625af5fe9acd398ec0c48a69ab52a79113d2c8d7d535547c5bd" exitCode=0 Aug 04 10:52:51.791042 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:51.791002 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg" Aug 04 10:52:51.791042 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:51.791008 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg" event={"ID":"2904dd2c-68f6-4e6a-9cf6-5063346046b8","Type":"ContainerDied","Data":"fa0c7b30f404b625af5fe9acd398ec0c48a69ab52a79113d2c8d7d535547c5bd"} Aug 04 10:52:51.791235 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:51.791046 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg" event={"ID":"2904dd2c-68f6-4e6a-9cf6-5063346046b8","Type":"ContainerDied","Data":"2d1f584609d84668de91a2fef97ef541058490be5748fcd4943768124a98ab35"} Aug 04 10:52:51.791235 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:51.791069 2580 scope.go:117] "RemoveContainer" containerID="fa0c7b30f404b625af5fe9acd398ec0c48a69ab52a79113d2c8d7d535547c5bd" Aug 04 10:52:51.799983 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:51.799964 2580 scope.go:117] "RemoveContainer" containerID="fa0c7b30f404b625af5fe9acd398ec0c48a69ab52a79113d2c8d7d535547c5bd" Aug 04 10:52:51.800243 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:52:51.800223 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa0c7b30f404b625af5fe9acd398ec0c48a69ab52a79113d2c8d7d535547c5bd\": container with ID starting with fa0c7b30f404b625af5fe9acd398ec0c48a69ab52a79113d2c8d7d535547c5bd not found: ID does not exist" containerID="fa0c7b30f404b625af5fe9acd398ec0c48a69ab52a79113d2c8d7d535547c5bd" Aug 04 10:52:51.800310 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:51.800257 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa0c7b30f404b625af5fe9acd398ec0c48a69ab52a79113d2c8d7d535547c5bd"} err="failed to get container status \"fa0c7b30f404b625af5fe9acd398ec0c48a69ab52a79113d2c8d7d535547c5bd\": rpc error: code = NotFound desc = could not find container \"fa0c7b30f404b625af5fe9acd398ec0c48a69ab52a79113d2c8d7d535547c5bd\": container with ID starting with fa0c7b30f404b625af5fe9acd398ec0c48a69ab52a79113d2c8d7d535547c5bd not found: ID does not exist" Aug 04 10:52:51.813950 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:51.813926 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg"] Aug 04 10:52:51.817704 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:51.817681 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-f9925d20-kserve-5fcd64548c-82stg"] Aug 04 10:52:52.858439 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:52.858394 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2" podUID="75a22383-40e4-4c42-9f05-036bc4db4576" containerName="main" containerID="cri-o://f312e4edc08bf81c823d545c3c52465f3fe4781be72e4e7be0762a18ddaceecf" gracePeriod=45 Aug 04 10:52:53.100909 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:53.100882 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2904dd2c-68f6-4e6a-9cf6-5063346046b8" path="/var/lib/kubelet/pods/2904dd2c-68f6-4e6a-9cf6-5063346046b8/volumes" Aug 04 10:52:53.106746 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:53.106725 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2" Aug 04 10:52:53.236329 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:53.236238 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75a22383-40e4-4c42-9f05-036bc4db4576-home\") pod \"75a22383-40e4-4c42-9f05-036bc4db4576\" (UID: \"75a22383-40e4-4c42-9f05-036bc4db4576\") " Aug 04 10:52:53.236329 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:53.236279 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75a22383-40e4-4c42-9f05-036bc4db4576-model-cache\") pod \"75a22383-40e4-4c42-9f05-036bc4db4576\" (UID: \"75a22383-40e4-4c42-9f05-036bc4db4576\") " Aug 04 10:52:53.236329 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:53.236323 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75a22383-40e4-4c42-9f05-036bc4db4576-tmp-dir\") pod \"75a22383-40e4-4c42-9f05-036bc4db4576\" (UID: \"75a22383-40e4-4c42-9f05-036bc4db4576\") " Aug 04 10:52:53.236622 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:53.236347 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75a22383-40e4-4c42-9f05-036bc4db4576-dshm\") pod \"75a22383-40e4-4c42-9f05-036bc4db4576\" (UID: \"75a22383-40e4-4c42-9f05-036bc4db4576\") " Aug 04 10:52:53.236622 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:53.236393 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkkmz\" (UniqueName: \"kubernetes.io/projected/75a22383-40e4-4c42-9f05-036bc4db4576-kube-api-access-rkkmz\") pod \"75a22383-40e4-4c42-9f05-036bc4db4576\" (UID: \"75a22383-40e4-4c42-9f05-036bc4db4576\") " Aug 04 10:52:53.236622 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:53.236438 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a22383-40e4-4c42-9f05-036bc4db4576-tls-certs\") pod \"75a22383-40e4-4c42-9f05-036bc4db4576\" (UID: \"75a22383-40e4-4c42-9f05-036bc4db4576\") " Aug 04 10:52:53.236791 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:53.236551 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a22383-40e4-4c42-9f05-036bc4db4576-home" (OuterVolumeSpecName: "home") pod "75a22383-40e4-4c42-9f05-036bc4db4576" (UID: "75a22383-40e4-4c42-9f05-036bc4db4576"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:52:53.236791 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:53.236658 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a22383-40e4-4c42-9f05-036bc4db4576-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "75a22383-40e4-4c42-9f05-036bc4db4576" (UID: "75a22383-40e4-4c42-9f05-036bc4db4576"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:52:53.236791 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:53.236669 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a22383-40e4-4c42-9f05-036bc4db4576-model-cache" (OuterVolumeSpecName: "model-cache") pod "75a22383-40e4-4c42-9f05-036bc4db4576" (UID: "75a22383-40e4-4c42-9f05-036bc4db4576"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:52:53.236791 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:53.236735 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75a22383-40e4-4c42-9f05-036bc4db4576-home\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:52:53.236791 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:53.236752 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75a22383-40e4-4c42-9f05-036bc4db4576-model-cache\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:52:53.236791 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:53.236766 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75a22383-40e4-4c42-9f05-036bc4db4576-tmp-dir\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:52:53.238709 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:53.238672 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75a22383-40e4-4c42-9f05-036bc4db4576-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "75a22383-40e4-4c42-9f05-036bc4db4576" (UID: "75a22383-40e4-4c42-9f05-036bc4db4576"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 10:52:53.238830 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:53.238746 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a22383-40e4-4c42-9f05-036bc4db4576-dshm" (OuterVolumeSpecName: "dshm") pod "75a22383-40e4-4c42-9f05-036bc4db4576" (UID: "75a22383-40e4-4c42-9f05-036bc4db4576"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:52:53.238830 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:53.238750 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a22383-40e4-4c42-9f05-036bc4db4576-kube-api-access-rkkmz" (OuterVolumeSpecName: "kube-api-access-rkkmz") pod "75a22383-40e4-4c42-9f05-036bc4db4576" (UID: "75a22383-40e4-4c42-9f05-036bc4db4576"). InnerVolumeSpecName "kube-api-access-rkkmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:52:53.337742 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:53.337702 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkkmz\" (UniqueName: \"kubernetes.io/projected/75a22383-40e4-4c42-9f05-036bc4db4576-kube-api-access-rkkmz\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:52:53.337742 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:53.337733 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a22383-40e4-4c42-9f05-036bc4db4576-tls-certs\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:52:53.337742 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:53.337743 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75a22383-40e4-4c42-9f05-036bc4db4576-dshm\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:52:53.802768 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:53.802735 2580 generic.go:356] "Generic (PLEG): container finished" podID="75a22383-40e4-4c42-9f05-036bc4db4576" containerID="f312e4edc08bf81c823d545c3c52465f3fe4781be72e4e7be0762a18ddaceecf" exitCode=0 Aug 04 10:52:53.802971 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:53.802782 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2" Aug 04 10:52:53.802971 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:53.802829 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2" event={"ID":"75a22383-40e4-4c42-9f05-036bc4db4576","Type":"ContainerDied","Data":"f312e4edc08bf81c823d545c3c52465f3fe4781be72e4e7be0762a18ddaceecf"} Aug 04 10:52:53.802971 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:53.802876 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2" event={"ID":"75a22383-40e4-4c42-9f05-036bc4db4576","Type":"ContainerDied","Data":"63e81617d9db14056bb6cfde2d2967abe48eece444c532915fd9e2262dddc434"} Aug 04 10:52:53.802971 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:53.802902 2580 scope.go:117] "RemoveContainer" containerID="f312e4edc08bf81c823d545c3c52465f3fe4781be72e4e7be0762a18ddaceecf" Aug 04 10:52:53.812506 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:53.812492 2580 scope.go:117] "RemoveContainer" containerID="f312e4edc08bf81c823d545c3c52465f3fe4781be72e4e7be0762a18ddaceecf" Aug 04 10:52:53.812827 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:52:53.812806 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f312e4edc08bf81c823d545c3c52465f3fe4781be72e4e7be0762a18ddaceecf\": container with ID starting with f312e4edc08bf81c823d545c3c52465f3fe4781be72e4e7be0762a18ddaceecf not found: ID does not exist" containerID="f312e4edc08bf81c823d545c3c52465f3fe4781be72e4e7be0762a18ddaceecf" Aug 04 10:52:53.812877 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:53.812837 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f312e4edc08bf81c823d545c3c52465f3fe4781be72e4e7be0762a18ddaceecf"} err="failed to get container status \"f312e4edc08bf81c823d545c3c52465f3fe4781be72e4e7be0762a18ddaceecf\": rpc error: code = NotFound desc = could not find container \"f312e4edc08bf81c823d545c3c52465f3fe4781be72e4e7be0762a18ddaceecf\": container with ID starting with f312e4edc08bf81c823d545c3c52465f3fe4781be72e4e7be0762a18ddaceecf not found: ID does not exist" Aug 04 10:52:53.824338 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:53.824311 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2"] Aug 04 10:52:53.827092 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:53.827063 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-f9925d20-kserve-69b9c79dc-fqhp2"] Aug 04 10:52:55.099877 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:55.099835 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75a22383-40e4-4c42-9f05-036bc4db4576" path="/var/lib/kubelet/pods/75a22383-40e4-4c42-9f05-036bc4db4576/volumes" Aug 04 10:52:55.432452 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:52:55.432417 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:53:05.330460 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:53:05.330374 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:53:12.333463 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:53:12.333424 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:53:19.551117 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:53:19.551084 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:53:30.028688 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:53:30.028656 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:53:36.637151 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:53:36.637115 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:53:49.240050 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:53:49.240005 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:53:54.434915 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:53:54.434875 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:54:12.032532 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:54:12.032498 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:54:17.236424 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:54:17.236387 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:54:30.530189 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:54:30.530153 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:54:35.638216 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:54:35.638179 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:54:42.136602 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:54:42.136550 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:54:47.235793 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:54:47.235749 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:54:53.536421 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:54:53.536384 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:55:03.331255 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:55:03.331220 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:55:12.341405 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:55:12.341372 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:55:30.750709 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:55:30.750668 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:55:38.034909 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:55:38.034869 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:55:48.028176 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:55:48.028142 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:56:03.130449 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:03.130368 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:56:09.037667 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:09.037634 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:56:15.437183 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:15.437144 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:56:27.037186 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:27.037150 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:56:35.536032 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:35.535993 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:56:40.143748 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:40.143709 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:56:49.292746 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.292713 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp"] Aug 04 10:56:49.293206 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.293045 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2904dd2c-68f6-4e6a-9cf6-5063346046b8" containerName="main" Aug 04 10:56:49.293206 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.293055 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2904dd2c-68f6-4e6a-9cf6-5063346046b8" containerName="main" Aug 04 10:56:49.293206 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.293068 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75a22383-40e4-4c42-9f05-036bc4db4576" containerName="main" Aug 04 10:56:49.293206 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.293074 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="75a22383-40e4-4c42-9f05-036bc4db4576" containerName="main" Aug 04 10:56:49.293206 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.293138 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2904dd2c-68f6-4e6a-9cf6-5063346046b8" containerName="main" Aug 04 10:56:49.293206 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.293148 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="75a22383-40e4-4c42-9f05-036bc4db4576" containerName="main" Aug 04 10:56:49.295936 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.295921 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:49.298743 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.298721 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5f7d6dc8-openshift-default-dockercfg-kqgpc\"" Aug 04 10:56:49.318231 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.318202 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp"] Aug 04 10:56:49.372529 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.372495 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4810695-e26c-4a4c-a02e-5ba1970307f2-credential-socket\") pod \"e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:49.372727 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.372552 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4810695-e26c-4a4c-a02e-5ba1970307f2-istio-data\") pod \"e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:49.372727 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.372604 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4810695-e26c-4a4c-a02e-5ba1970307f2-istio-podinfo\") pod \"e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:49.372727 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.372667 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4810695-e26c-4a4c-a02e-5ba1970307f2-istiod-ca-cert\") pod \"e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:49.372869 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.372724 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4810695-e26c-4a4c-a02e-5ba1970307f2-workload-socket\") pod \"e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:49.372869 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.372751 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4810695-e26c-4a4c-a02e-5ba1970307f2-istio-token\") pod \"e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:49.372869 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.372778 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4810695-e26c-4a4c-a02e-5ba1970307f2-workload-certs\") pod \"e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:49.372869 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.372810 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r77qb\" (UniqueName: \"kubernetes.io/projected/a4810695-e26c-4a4c-a02e-5ba1970307f2-kube-api-access-r77qb\") pod \"e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:49.373046 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.372884 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4810695-e26c-4a4c-a02e-5ba1970307f2-istio-envoy\") pod \"e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:49.473692 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.473657 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4810695-e26c-4a4c-a02e-5ba1970307f2-istio-envoy\") pod \"e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:49.473932 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.473748 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4810695-e26c-4a4c-a02e-5ba1970307f2-credential-socket\") pod \"e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:49.473932 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.473781 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4810695-e26c-4a4c-a02e-5ba1970307f2-istio-data\") pod \"e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:49.473932 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.473803 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4810695-e26c-4a4c-a02e-5ba1970307f2-istio-podinfo\") pod \"e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:49.473932 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.473833 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4810695-e26c-4a4c-a02e-5ba1970307f2-istiod-ca-cert\") pod \"e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:49.473932 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.473865 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4810695-e26c-4a4c-a02e-5ba1970307f2-workload-socket\") pod \"e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:49.473932 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.473889 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4810695-e26c-4a4c-a02e-5ba1970307f2-istio-token\") pod \"e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:49.473932 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.473918 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4810695-e26c-4a4c-a02e-5ba1970307f2-workload-certs\") pod \"e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:49.474280 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.473944 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r77qb\" (UniqueName: \"kubernetes.io/projected/a4810695-e26c-4a4c-a02e-5ba1970307f2-kube-api-access-r77qb\") pod \"e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:49.477733 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.474597 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4810695-e26c-4a4c-a02e-5ba1970307f2-credential-socket\") pod \"e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:49.477733 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.474621 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4810695-e26c-4a4c-a02e-5ba1970307f2-istio-data\") pod \"e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:49.477733 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.474870 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4810695-e26c-4a4c-a02e-5ba1970307f2-workload-certs\") pod \"e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:49.477733 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.475025 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4810695-e26c-4a4c-a02e-5ba1970307f2-workload-socket\") pod \"e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:49.477733 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.475170 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4810695-e26c-4a4c-a02e-5ba1970307f2-istiod-ca-cert\") pod \"e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:49.477733 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.477229 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4810695-e26c-4a4c-a02e-5ba1970307f2-istio-podinfo\") pod \"e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:49.482954 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.482931 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4810695-e26c-4a4c-a02e-5ba1970307f2-istio-envoy\") pod \"e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:49.487055 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.487031 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4810695-e26c-4a4c-a02e-5ba1970307f2-istio-token\") pod \"e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:49.487228 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.487211 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r77qb\" (UniqueName: \"kubernetes.io/projected/a4810695-e26c-4a4c-a02e-5ba1970307f2-kube-api-access-r77qb\") pod \"e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:49.605680 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.605637 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:49.766046 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.766014 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp"] Aug 04 10:56:49.769054 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:56:49.769028 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4810695_e26c_4a4c_a02e_5ba1970307f2.slice/crio-ba524c48c43891ef59b40e41237776eba3649f8b847f4e84d653ebfce3073ec3 WatchSource:0}: Error finding container ba524c48c43891ef59b40e41237776eba3649f8b847f4e84d653ebfce3073ec3: Status 404 returned error can't find the container with id ba524c48c43891ef59b40e41237776eba3649f8b847f4e84d653ebfce3073ec3 Aug 04 10:56:49.771203 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.771160 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:56:49.771297 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.771241 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:56:49.771297 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.771285 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:56:49.882904 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.882801 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" event={"ID":"a4810695-e26c-4a4c-a02e-5ba1970307f2","Type":"ContainerStarted","Data":"22019b8e7dca8dd9701cfc3175d607fa20acda5d1ebf6c67de9dab7bbbfc42d0"} Aug 04 10:56:49.882904 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.882873 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" event={"ID":"a4810695-e26c-4a4c-a02e-5ba1970307f2","Type":"ContainerStarted","Data":"ba524c48c43891ef59b40e41237776eba3649f8b847f4e84d653ebfce3073ec3"} Aug 04 10:56:49.907036 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:49.906977 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" podStartSLOduration=0.906958682 podStartE2EDuration="906.958682ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:56:49.905516204 +0000 UTC m=+1749.326403650" watchObservedRunningTime="2026-08-04 10:56:49.906958682 +0000 UTC m=+1749.327846092" Aug 04 10:56:50.606114 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:50.606076 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:50.610674 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:50.610648 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:50.888104 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:50.888017 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:50.889119 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:50.889100 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:56:55.036807 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:56:55.036768 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:57:04.955000 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:04.954908 2580 generic.go:356] "Generic (PLEG): container finished" podID="a4810695-e26c-4a4c-a02e-5ba1970307f2" containerID="22019b8e7dca8dd9701cfc3175d607fa20acda5d1ebf6c67de9dab7bbbfc42d0" exitCode=0 Aug 04 10:57:04.955000 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:04.954975 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" event={"ID":"a4810695-e26c-4a4c-a02e-5ba1970307f2","Type":"ContainerDied","Data":"22019b8e7dca8dd9701cfc3175d607fa20acda5d1ebf6c67de9dab7bbbfc42d0"} Aug 04 10:57:04.955409 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:04.955350 2580 scope.go:117] "RemoveContainer" containerID="22019b8e7dca8dd9701cfc3175d607fa20acda5d1ebf6c67de9dab7bbbfc42d0" Aug 04 10:57:04.956539 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:04.956505 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:57:04.956634 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:04.956615 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:57:04.956677 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:04.956663 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:57:05.888842 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:05.888806 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:57:05.962991 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:05.962951 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" event={"ID":"a4810695-e26c-4a4c-a02e-5ba1970307f2","Type":"ContainerStarted","Data":"c63c235091f09edfe361decd366f4cc80f5cad7336f02b6caceadac2ed13d66c"} Aug 04 10:57:06.606578 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:06.606538 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:57:06.607788 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:06.607764 2580 patch_prober.go:28] interesting pod/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Aug 04 10:57:06.607906 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:06.607810 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" podUID="a4810695-e26c-4a4c-a02e-5ba1970307f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Aug 04 10:57:07.606772 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:07.606722 2580 patch_prober.go:28] interesting pod/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Aug 04 10:57:07.607192 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:07.606801 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" podUID="a4810695-e26c-4a4c-a02e-5ba1970307f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Aug 04 10:57:08.606992 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:08.606949 2580 patch_prober.go:28] interesting pod/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Aug 04 10:57:08.607387 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:08.607016 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" podUID="a4810695-e26c-4a4c-a02e-5ba1970307f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Aug 04 10:57:09.606153 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:09.606112 2580 patch_prober.go:28] interesting pod/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Aug 04 10:57:09.606333 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:09.606176 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" podUID="a4810695-e26c-4a4c-a02e-5ba1970307f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Aug 04 10:57:10.606216 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:10.606173 2580 patch_prober.go:28] interesting pod/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Aug 04 10:57:10.606701 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:10.606252 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" podUID="a4810695-e26c-4a4c-a02e-5ba1970307f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Aug 04 10:57:11.611403 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:11.611368 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:57:11.611919 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:11.611897 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:57:11.612706 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:11.612686 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:57:23.444891 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:23.444856 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp"] Aug 04 10:57:23.445354 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:23.445116 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" podUID="a4810695-e26c-4a4c-a02e-5ba1970307f2" containerName="istio-proxy" containerID="cri-o://c63c235091f09edfe361decd366f4cc80f5cad7336f02b6caceadac2ed13d66c" gracePeriod=30 Aug 04 10:57:24.501206 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.501173 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q"] Aug 04 10:57:24.504675 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.504655 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:24.508095 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.508074 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-03e71fbe-gw-openshift-default-dockercfg-jk2nv\"" Aug 04 10:57:24.522121 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.522083 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q"] Aug 04 10:57:24.594124 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.594087 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dcd5d52-978a-439f-9aa4-a243bc497147-istio-envoy\") pod \"e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:24.594124 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.594126 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dcd5d52-978a-439f-9aa4-a243bc497147-istio-token\") pod \"e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:24.594382 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.594158 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dcd5d52-978a-439f-9aa4-a243bc497147-istiod-ca-cert\") pod \"e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:24.594382 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.594181 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dcd5d52-978a-439f-9aa4-a243bc497147-credential-socket\") pod \"e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:24.594382 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.594211 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dcd5d52-978a-439f-9aa4-a243bc497147-workload-certs\") pod \"e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:24.594382 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.594250 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dcd5d52-978a-439f-9aa4-a243bc497147-istio-data\") pod \"e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:24.594382 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.594274 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dcd5d52-978a-439f-9aa4-a243bc497147-istio-podinfo\") pod \"e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:24.594382 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.594311 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nspx9\" (UniqueName: \"kubernetes.io/projected/2dcd5d52-978a-439f-9aa4-a243bc497147-kube-api-access-nspx9\") pod \"e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:24.594382 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.594339 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dcd5d52-978a-439f-9aa4-a243bc497147-workload-socket\") pod \"e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:24.695162 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.695124 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dcd5d52-978a-439f-9aa4-a243bc497147-istio-data\") pod \"e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:24.695162 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.695168 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dcd5d52-978a-439f-9aa4-a243bc497147-istio-podinfo\") pod \"e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:24.695419 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.695201 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nspx9\" (UniqueName: \"kubernetes.io/projected/2dcd5d52-978a-439f-9aa4-a243bc497147-kube-api-access-nspx9\") pod \"e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:24.695419 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.695226 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dcd5d52-978a-439f-9aa4-a243bc497147-workload-socket\") pod \"e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:24.695419 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.695243 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dcd5d52-978a-439f-9aa4-a243bc497147-istio-envoy\") pod \"e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:24.695419 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.695258 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dcd5d52-978a-439f-9aa4-a243bc497147-istio-token\") pod \"e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:24.695419 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.695279 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dcd5d52-978a-439f-9aa4-a243bc497147-istiod-ca-cert\") pod \"e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:24.695419 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.695304 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dcd5d52-978a-439f-9aa4-a243bc497147-credential-socket\") pod \"e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:24.695727 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.695545 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dcd5d52-978a-439f-9aa4-a243bc497147-workload-certs\") pod \"e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:24.695727 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.695615 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dcd5d52-978a-439f-9aa4-a243bc497147-istio-data\") pod \"e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:24.695854 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.695831 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dcd5d52-978a-439f-9aa4-a243bc497147-workload-socket\") pod \"e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:24.695973 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.695951 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dcd5d52-978a-439f-9aa4-a243bc497147-credential-socket\") pod \"e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:24.696037 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.695993 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dcd5d52-978a-439f-9aa4-a243bc497147-workload-certs\") pod \"e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:24.696333 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.696282 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dcd5d52-978a-439f-9aa4-a243bc497147-istiod-ca-cert\") pod \"e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:24.697838 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.697812 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dcd5d52-978a-439f-9aa4-a243bc497147-istio-envoy\") pod \"e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:24.698192 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.698170 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dcd5d52-978a-439f-9aa4-a243bc497147-istio-podinfo\") pod \"e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:24.704782 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.704760 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dcd5d52-978a-439f-9aa4-a243bc497147-istio-token\") pod \"e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:24.705278 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.705258 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nspx9\" (UniqueName: \"kubernetes.io/projected/2dcd5d52-978a-439f-9aa4-a243bc497147-kube-api-access-nspx9\") pod \"e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:24.814796 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.814702 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:24.974613 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.974544 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q"] Aug 04 10:57:24.976956 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:57:24.976912 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dcd5d52_978a_439f_9aa4_a243bc497147.slice/crio-93497443bae5f873de0bb2a356e3992e13b09da4cf18824ab7cacd939aede79b WatchSource:0}: Error finding container 93497443bae5f873de0bb2a356e3992e13b09da4cf18824ab7cacd939aede79b: Status 404 returned error can't find the container with id 93497443bae5f873de0bb2a356e3992e13b09da4cf18824ab7cacd939aede79b Aug 04 10:57:24.979455 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.979419 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:57:24.979603 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.979508 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:57:24.979603 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:24.979542 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:57:25.069812 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:25.069728 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" event={"ID":"2dcd5d52-978a-439f-9aa4-a243bc497147","Type":"ContainerStarted","Data":"c5908a46e81d23bf4fdfd73faea55edf84bc6e7c6212de5d480bc5a229c47bab"} Aug 04 10:57:25.069812 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:25.069782 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" event={"ID":"2dcd5d52-978a-439f-9aa4-a243bc497147","Type":"ContainerStarted","Data":"93497443bae5f873de0bb2a356e3992e13b09da4cf18824ab7cacd939aede79b"} Aug 04 10:57:25.091316 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:25.091253 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" podStartSLOduration=1.091231973 podStartE2EDuration="1.091231973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:57:25.090457378 +0000 UTC m=+1784.511344792" watchObservedRunningTime="2026-08-04 10:57:25.091231973 +0000 UTC m=+1784.512119386" Aug 04 10:57:25.814955 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:25.814913 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:26.240504 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:26.240469 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:57:26.613951 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:26.613911 2580 patch_prober.go:28] interesting pod/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 10:57:26.614139 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:26.613972 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" podUID="a4810695-e26c-4a4c-a02e-5ba1970307f2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 10:57:26.820259 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:26.820230 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:27.081304 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:27.081224 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:27.082268 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:27.082248 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:28.701476 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:28.701449 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:57:28.729114 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:28.729031 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4810695-e26c-4a4c-a02e-5ba1970307f2-workload-certs\") pod \"a4810695-e26c-4a4c-a02e-5ba1970307f2\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " Aug 04 10:57:28.729114 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:28.729083 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4810695-e26c-4a4c-a02e-5ba1970307f2-istio-envoy\") pod \"a4810695-e26c-4a4c-a02e-5ba1970307f2\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " Aug 04 10:57:28.729330 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:28.729118 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4810695-e26c-4a4c-a02e-5ba1970307f2-istio-podinfo\") pod \"a4810695-e26c-4a4c-a02e-5ba1970307f2\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " Aug 04 10:57:28.729330 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:28.729172 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4810695-e26c-4a4c-a02e-5ba1970307f2-istio-data\") pod \"a4810695-e26c-4a4c-a02e-5ba1970307f2\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " Aug 04 10:57:28.729330 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:28.729197 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r77qb\" (UniqueName: \"kubernetes.io/projected/a4810695-e26c-4a4c-a02e-5ba1970307f2-kube-api-access-r77qb\") pod \"a4810695-e26c-4a4c-a02e-5ba1970307f2\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " Aug 04 10:57:28.729330 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:28.729237 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4810695-e26c-4a4c-a02e-5ba1970307f2-credential-socket\") pod \"a4810695-e26c-4a4c-a02e-5ba1970307f2\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " Aug 04 10:57:28.729330 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:28.729279 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4810695-e26c-4a4c-a02e-5ba1970307f2-istiod-ca-cert\") pod \"a4810695-e26c-4a4c-a02e-5ba1970307f2\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " Aug 04 10:57:28.729610 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:28.729338 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4810695-e26c-4a4c-a02e-5ba1970307f2-workload-socket\") pod \"a4810695-e26c-4a4c-a02e-5ba1970307f2\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " Aug 04 10:57:28.729610 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:28.729353 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4810695-e26c-4a4c-a02e-5ba1970307f2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a4810695-e26c-4a4c-a02e-5ba1970307f2" (UID: "a4810695-e26c-4a4c-a02e-5ba1970307f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:57:28.729610 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:28.729388 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4810695-e26c-4a4c-a02e-5ba1970307f2-istio-token\") pod \"a4810695-e26c-4a4c-a02e-5ba1970307f2\" (UID: \"a4810695-e26c-4a4c-a02e-5ba1970307f2\") " Aug 04 10:57:28.729777 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:28.729627 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4810695-e26c-4a4c-a02e-5ba1970307f2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a4810695-e26c-4a4c-a02e-5ba1970307f2" (UID: "a4810695-e26c-4a4c-a02e-5ba1970307f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:57:28.729777 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:28.729676 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4810695-e26c-4a4c-a02e-5ba1970307f2-workload-certs\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:57:28.729884 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:28.729806 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4810695-e26c-4a4c-a02e-5ba1970307f2-istio-data" (OuterVolumeSpecName: "istio-data") pod "a4810695-e26c-4a4c-a02e-5ba1970307f2" (UID: "a4810695-e26c-4a4c-a02e-5ba1970307f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:57:28.730361 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:28.729985 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4810695-e26c-4a4c-a02e-5ba1970307f2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a4810695-e26c-4a4c-a02e-5ba1970307f2" (UID: "a4810695-e26c-4a4c-a02e-5ba1970307f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 10:57:28.730361 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:28.730312 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4810695-e26c-4a4c-a02e-5ba1970307f2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a4810695-e26c-4a4c-a02e-5ba1970307f2" (UID: "a4810695-e26c-4a4c-a02e-5ba1970307f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:57:28.731849 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:28.731806 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4810695-e26c-4a4c-a02e-5ba1970307f2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a4810695-e26c-4a4c-a02e-5ba1970307f2" (UID: "a4810695-e26c-4a4c-a02e-5ba1970307f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 10:57:28.732064 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:28.732044 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4810695-e26c-4a4c-a02e-5ba1970307f2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a4810695-e26c-4a4c-a02e-5ba1970307f2" (UID: "a4810695-e26c-4a4c-a02e-5ba1970307f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:57:28.732309 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:28.732294 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4810695-e26c-4a4c-a02e-5ba1970307f2-kube-api-access-r77qb" (OuterVolumeSpecName: "kube-api-access-r77qb") pod "a4810695-e26c-4a4c-a02e-5ba1970307f2" (UID: "a4810695-e26c-4a4c-a02e-5ba1970307f2"). InnerVolumeSpecName "kube-api-access-r77qb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:57:28.732309 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:28.732314 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4810695-e26c-4a4c-a02e-5ba1970307f2-istio-token" (OuterVolumeSpecName: "istio-token") pod "a4810695-e26c-4a4c-a02e-5ba1970307f2" (UID: "a4810695-e26c-4a4c-a02e-5ba1970307f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:57:28.831094 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:28.831059 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4810695-e26c-4a4c-a02e-5ba1970307f2-istio-data\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:57:28.831094 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:28.831086 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r77qb\" (UniqueName: \"kubernetes.io/projected/a4810695-e26c-4a4c-a02e-5ba1970307f2-kube-api-access-r77qb\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:57:28.831094 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:28.831095 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4810695-e26c-4a4c-a02e-5ba1970307f2-credential-socket\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:57:28.831094 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:28.831105 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4810695-e26c-4a4c-a02e-5ba1970307f2-istiod-ca-cert\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:57:28.831350 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:28.831113 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4810695-e26c-4a4c-a02e-5ba1970307f2-workload-socket\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:57:28.831350 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:28.831121 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4810695-e26c-4a4c-a02e-5ba1970307f2-istio-token\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:57:28.831350 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:28.831130 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4810695-e26c-4a4c-a02e-5ba1970307f2-istio-envoy\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:57:28.831350 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:28.831138 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4810695-e26c-4a4c-a02e-5ba1970307f2-istio-podinfo\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:57:29.091108 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:29.091025 2580 generic.go:356] "Generic (PLEG): container finished" podID="a4810695-e26c-4a4c-a02e-5ba1970307f2" containerID="c63c235091f09edfe361decd366f4cc80f5cad7336f02b6caceadac2ed13d66c" exitCode=0 Aug 04 10:57:29.091108 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:29.091080 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" Aug 04 10:57:29.091276 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:29.091117 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" event={"ID":"a4810695-e26c-4a4c-a02e-5ba1970307f2","Type":"ContainerDied","Data":"c63c235091f09edfe361decd366f4cc80f5cad7336f02b6caceadac2ed13d66c"} Aug 04 10:57:29.091276 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:29.091184 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp" event={"ID":"a4810695-e26c-4a4c-a02e-5ba1970307f2","Type":"ContainerDied","Data":"ba524c48c43891ef59b40e41237776eba3649f8b847f4e84d653ebfce3073ec3"} Aug 04 10:57:29.091276 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:29.091209 2580 scope.go:117] "RemoveContainer" containerID="c63c235091f09edfe361decd366f4cc80f5cad7336f02b6caceadac2ed13d66c" Aug 04 10:57:29.103606 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:29.101891 2580 scope.go:117] "RemoveContainer" containerID="22019b8e7dca8dd9701cfc3175d607fa20acda5d1ebf6c67de9dab7bbbfc42d0" Aug 04 10:57:29.111798 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:29.111780 2580 scope.go:117] "RemoveContainer" containerID="c63c235091f09edfe361decd366f4cc80f5cad7336f02b6caceadac2ed13d66c" Aug 04 10:57:29.112074 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:57:29.112055 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c63c235091f09edfe361decd366f4cc80f5cad7336f02b6caceadac2ed13d66c\": container with ID starting with c63c235091f09edfe361decd366f4cc80f5cad7336f02b6caceadac2ed13d66c not found: ID does not exist" containerID="c63c235091f09edfe361decd366f4cc80f5cad7336f02b6caceadac2ed13d66c" Aug 04 10:57:29.112117 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:29.112085 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c63c235091f09edfe361decd366f4cc80f5cad7336f02b6caceadac2ed13d66c"} err="failed to get container status \"c63c235091f09edfe361decd366f4cc80f5cad7336f02b6caceadac2ed13d66c\": rpc error: code = NotFound desc = could not find container \"c63c235091f09edfe361decd366f4cc80f5cad7336f02b6caceadac2ed13d66c\": container with ID starting with c63c235091f09edfe361decd366f4cc80f5cad7336f02b6caceadac2ed13d66c not found: ID does not exist" Aug 04 10:57:29.112117 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:29.112103 2580 scope.go:117] "RemoveContainer" containerID="22019b8e7dca8dd9701cfc3175d607fa20acda5d1ebf6c67de9dab7bbbfc42d0" Aug 04 10:57:29.112397 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:57:29.112361 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22019b8e7dca8dd9701cfc3175d607fa20acda5d1ebf6c67de9dab7bbbfc42d0\": container with ID starting with 22019b8e7dca8dd9701cfc3175d607fa20acda5d1ebf6c67de9dab7bbbfc42d0 not found: ID does not exist" containerID="22019b8e7dca8dd9701cfc3175d607fa20acda5d1ebf6c67de9dab7bbbfc42d0" Aug 04 10:57:29.112443 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:29.112403 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22019b8e7dca8dd9701cfc3175d607fa20acda5d1ebf6c67de9dab7bbbfc42d0"} err="failed to get container status \"22019b8e7dca8dd9701cfc3175d607fa20acda5d1ebf6c67de9dab7bbbfc42d0\": rpc error: code = NotFound desc = could not find container \"22019b8e7dca8dd9701cfc3175d607fa20acda5d1ebf6c67de9dab7bbbfc42d0\": container with ID starting with 22019b8e7dca8dd9701cfc3175d607fa20acda5d1ebf6c67de9dab7bbbfc42d0 not found: ID does not exist" Aug 04 10:57:29.122784 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:29.122755 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp"] Aug 04 10:57:29.133605 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:29.133578 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5f7d6dc8-openshift-default-68fcf46bd6-5ddzp"] Aug 04 10:57:31.107275 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:31.107236 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4810695-e26c-4a4c-a02e-5ba1970307f2" path="/var/lib/kubelet/pods/a4810695-e26c-4a4c-a02e-5ba1970307f2/volumes" Aug 04 10:57:33.855641 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:33.855606 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:57:39.143261 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:39.143229 2580 generic.go:356] "Generic (PLEG): container finished" podID="2dcd5d52-978a-439f-9aa4-a243bc497147" containerID="c5908a46e81d23bf4fdfd73faea55edf84bc6e7c6212de5d480bc5a229c47bab" exitCode=0 Aug 04 10:57:39.143706 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:39.143310 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" event={"ID":"2dcd5d52-978a-439f-9aa4-a243bc497147","Type":"ContainerDied","Data":"c5908a46e81d23bf4fdfd73faea55edf84bc6e7c6212de5d480bc5a229c47bab"} Aug 04 10:57:39.143761 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:39.143713 2580 scope.go:117] "RemoveContainer" containerID="c5908a46e81d23bf4fdfd73faea55edf84bc6e7c6212de5d480bc5a229c47bab" Aug 04 10:57:39.144686 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:39.144654 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:57:39.144745 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:39.144729 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:57:39.144779 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:39.144766 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:57:40.149639 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:40.149601 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" event={"ID":"2dcd5d52-978a-439f-9aa4-a243bc497147","Type":"ContainerStarted","Data":"a656586df1456762457d9b0ab6b49120cf2086b16cdc07816c05b05226409aa9"} Aug 04 10:57:40.815312 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:40.815270 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:40.816466 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:40.816441 2580 patch_prober.go:28] interesting pod/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Aug 04 10:57:40.816531 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:40.816493 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" podUID="2dcd5d52-978a-439f-9aa4-a243bc497147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Aug 04 10:57:41.130482 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:41.130456 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tb4zk_c5922147-cb42-4f62-9880-8deaa51a4b0f/ovn-acl-logging/0.log" Aug 04 10:57:41.136206 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:41.136182 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tb4zk_c5922147-cb42-4f62-9880-8deaa51a4b0f/ovn-acl-logging/0.log" Aug 04 10:57:41.815789 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:41.815749 2580 patch_prober.go:28] interesting pod/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Aug 04 10:57:41.816202 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:41.815821 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" podUID="2dcd5d52-978a-439f-9aa4-a243bc497147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Aug 04 10:57:42.081982 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:42.081863 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:42.815370 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:42.815331 2580 patch_prober.go:28] interesting pod/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Aug 04 10:57:42.815554 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:42.815393 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" podUID="2dcd5d52-978a-439f-9aa4-a243bc497147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Aug 04 10:57:43.815137 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:43.815098 2580 patch_prober.go:28] interesting pod/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Aug 04 10:57:43.815547 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:43.815168 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" podUID="2dcd5d52-978a-439f-9aa4-a243bc497147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Aug 04 10:57:44.815639 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:44.815600 2580 patch_prober.go:28] interesting pod/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Aug 04 10:57:44.816045 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:44.815660 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" podUID="2dcd5d52-978a-439f-9aa4-a243bc497147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Aug 04 10:57:45.819907 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:45.819873 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:45.820887 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:45.820870 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:57:58.589188 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:58.589143 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q"] Aug 04 10:57:58.589741 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:58.589695 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" podUID="2dcd5d52-978a-439f-9aa4-a243bc497147" containerName="istio-proxy" containerID="cri-o://a656586df1456762457d9b0ab6b49120cf2086b16cdc07816c05b05226409aa9" gracePeriod=30 Aug 04 10:57:59.507316 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.507276 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq"] Aug 04 10:57:59.507682 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.507669 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4810695-e26c-4a4c-a02e-5ba1970307f2" containerName="istio-proxy" Aug 04 10:57:59.507732 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.507685 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4810695-e26c-4a4c-a02e-5ba1970307f2" containerName="istio-proxy" Aug 04 10:57:59.507732 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.507697 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4810695-e26c-4a4c-a02e-5ba1970307f2" containerName="istio-proxy" Aug 04 10:57:59.507732 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.507703 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4810695-e26c-4a4c-a02e-5ba1970307f2" containerName="istio-proxy" Aug 04 10:57:59.507824 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.507761 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4810695-e26c-4a4c-a02e-5ba1970307f2" containerName="istio-proxy" Aug 04 10:57:59.507824 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.507771 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4810695-e26c-4a4c-a02e-5ba1970307f2" containerName="istio-proxy" Aug 04 10:57:59.511963 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.511940 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:57:59.515133 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.515087 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-fb58dd84-openshift-default-dockercfg-95kt2\"" Aug 04 10:57:59.543626 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.543591 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq"] Aug 04 10:57:59.597370 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.597329 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4246b32-957a-4473-b784-5fc5174e24e2-istio-token\") pod \"e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:57:59.597870 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.597384 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4246b32-957a-4473-b784-5fc5174e24e2-workload-certs\") pod \"e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:57:59.597870 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.597450 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4246b32-957a-4473-b784-5fc5174e24e2-istiod-ca-cert\") pod \"e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:57:59.597870 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.597597 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4246b32-957a-4473-b784-5fc5174e24e2-credential-socket\") pod \"e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:57:59.597870 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.597646 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4246b32-957a-4473-b784-5fc5174e24e2-istio-envoy\") pod \"e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:57:59.597870 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.597697 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4246b32-957a-4473-b784-5fc5174e24e2-workload-socket\") pod \"e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:57:59.597870 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.597734 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4246b32-957a-4473-b784-5fc5174e24e2-istio-data\") pod \"e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:57:59.597870 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.597759 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4246b32-957a-4473-b784-5fc5174e24e2-istio-podinfo\") pod \"e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:57:59.597870 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.597788 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5bbp\" (UniqueName: \"kubernetes.io/projected/a4246b32-957a-4473-b784-5fc5174e24e2-kube-api-access-w5bbp\") pod \"e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:57:59.698835 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.698794 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4246b32-957a-4473-b784-5fc5174e24e2-istiod-ca-cert\") pod \"e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:57:59.699020 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.698866 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4246b32-957a-4473-b784-5fc5174e24e2-credential-socket\") pod \"e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:57:59.699020 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.698901 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4246b32-957a-4473-b784-5fc5174e24e2-istio-envoy\") pod \"e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:57:59.699020 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.698944 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4246b32-957a-4473-b784-5fc5174e24e2-workload-socket\") pod \"e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:57:59.699020 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.698975 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4246b32-957a-4473-b784-5fc5174e24e2-istio-data\") pod \"e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:57:59.699020 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.698999 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4246b32-957a-4473-b784-5fc5174e24e2-istio-podinfo\") pod \"e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:57:59.699293 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.699028 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5bbp\" (UniqueName: \"kubernetes.io/projected/a4246b32-957a-4473-b784-5fc5174e24e2-kube-api-access-w5bbp\") pod \"e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:57:59.699293 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.699059 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4246b32-957a-4473-b784-5fc5174e24e2-istio-token\") pod \"e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:57:59.699293 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.699091 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4246b32-957a-4473-b784-5fc5174e24e2-workload-certs\") pod \"e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:57:59.699438 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.699374 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4246b32-957a-4473-b784-5fc5174e24e2-credential-socket\") pod \"e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:57:59.699438 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.699396 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4246b32-957a-4473-b784-5fc5174e24e2-workload-socket\") pod \"e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:57:59.699633 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.699594 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4246b32-957a-4473-b784-5fc5174e24e2-istiod-ca-cert\") pod \"e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:57:59.699856 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.699837 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4246b32-957a-4473-b784-5fc5174e24e2-workload-certs\") pod \"e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:57:59.699957 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.699859 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4246b32-957a-4473-b784-5fc5174e24e2-istio-data\") pod \"e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:57:59.701417 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.701390 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4246b32-957a-4473-b784-5fc5174e24e2-istio-envoy\") pod \"e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:57:59.701712 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.701693 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4246b32-957a-4473-b784-5fc5174e24e2-istio-podinfo\") pod \"e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:57:59.708113 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.708087 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4246b32-957a-4473-b784-5fc5174e24e2-istio-token\") pod \"e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:57:59.708248 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.708226 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5bbp\" (UniqueName: \"kubernetes.io/projected/a4246b32-957a-4473-b784-5fc5174e24e2-kube-api-access-w5bbp\") pod \"e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:57:59.822418 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.822329 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:57:59.973320 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.973294 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq"] Aug 04 10:57:59.975089 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:57:59.975052 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4246b32_957a_4473_b784_5fc5174e24e2.slice/crio-a7674e72ec4b5151c2a8181050f04b100fea35afbdca9892be6ea8f4067e9da5 WatchSource:0}: Error finding container a7674e72ec4b5151c2a8181050f04b100fea35afbdca9892be6ea8f4067e9da5: Status 404 returned error can't find the container with id a7674e72ec4b5151c2a8181050f04b100fea35afbdca9892be6ea8f4067e9da5 Aug 04 10:57:59.977306 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.977268 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:57:59.977417 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.977352 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:57:59.977417 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:57:59.977397 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:58:00.245169 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:00.245129 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" event={"ID":"a4246b32-957a-4473-b784-5fc5174e24e2","Type":"ContainerStarted","Data":"5dc5ea4157ba4c2e6ec55ade6d767a9ed0d7be340beb9e4f7cc02a0496741537"} Aug 04 10:58:00.245379 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:00.245182 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" event={"ID":"a4246b32-957a-4473-b784-5fc5174e24e2","Type":"ContainerStarted","Data":"a7674e72ec4b5151c2a8181050f04b100fea35afbdca9892be6ea8f4067e9da5"} Aug 04 10:58:00.269465 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:00.269397 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" podStartSLOduration=1.269378519 podStartE2EDuration="1.269378519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:58:00.267416076 +0000 UTC m=+1819.688303487" watchObservedRunningTime="2026-08-04 10:58:00.269378519 +0000 UTC m=+1819.690265930" Aug 04 10:58:00.822415 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:00.822377 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:58:00.823312 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:00.823281 2580 patch_prober.go:28] interesting pod/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 10:58:00.823377 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:00.823351 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" podUID="2dcd5d52-978a-439f-9aa4-a243bc497147" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 10:58:01.438328 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:01.438292 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:58:01.828265 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:01.828189 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:58:02.256035 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:02.256003 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:58:02.257011 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:02.256990 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:58:03.849628 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:03.849605 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:58:03.942920 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:03.942833 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dcd5d52-978a-439f-9aa4-a243bc497147-workload-socket\") pod \"2dcd5d52-978a-439f-9aa4-a243bc497147\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " Aug 04 10:58:03.942920 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:03.942891 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dcd5d52-978a-439f-9aa4-a243bc497147-credential-socket\") pod \"2dcd5d52-978a-439f-9aa4-a243bc497147\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " Aug 04 10:58:03.942920 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:03.942913 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dcd5d52-978a-439f-9aa4-a243bc497147-istio-envoy\") pod \"2dcd5d52-978a-439f-9aa4-a243bc497147\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " Aug 04 10:58:03.943209 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:03.942932 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nspx9\" (UniqueName: \"kubernetes.io/projected/2dcd5d52-978a-439f-9aa4-a243bc497147-kube-api-access-nspx9\") pod \"2dcd5d52-978a-439f-9aa4-a243bc497147\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " Aug 04 10:58:03.943209 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:03.942971 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dcd5d52-978a-439f-9aa4-a243bc497147-istio-podinfo\") pod \"2dcd5d52-978a-439f-9aa4-a243bc497147\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " Aug 04 10:58:03.943209 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:03.942990 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dcd5d52-978a-439f-9aa4-a243bc497147-istio-token\") pod \"2dcd5d52-978a-439f-9aa4-a243bc497147\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " Aug 04 10:58:03.943209 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:03.943003 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dcd5d52-978a-439f-9aa4-a243bc497147-workload-certs\") pod \"2dcd5d52-978a-439f-9aa4-a243bc497147\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " Aug 04 10:58:03.943209 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:03.943146 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dcd5d52-978a-439f-9aa4-a243bc497147-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2dcd5d52-978a-439f-9aa4-a243bc497147" (UID: "2dcd5d52-978a-439f-9aa4-a243bc497147"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:58:03.943209 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:03.943157 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dcd5d52-978a-439f-9aa4-a243bc497147-istiod-ca-cert\") pod \"2dcd5d52-978a-439f-9aa4-a243bc497147\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " Aug 04 10:58:03.943209 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:03.943190 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dcd5d52-978a-439f-9aa4-a243bc497147-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2dcd5d52-978a-439f-9aa4-a243bc497147" (UID: "2dcd5d52-978a-439f-9aa4-a243bc497147"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:58:03.943554 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:03.943213 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dcd5d52-978a-439f-9aa4-a243bc497147-istio-data\") pod \"2dcd5d52-978a-439f-9aa4-a243bc497147\" (UID: \"2dcd5d52-978a-439f-9aa4-a243bc497147\") " Aug 04 10:58:03.943554 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:03.943319 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dcd5d52-978a-439f-9aa4-a243bc497147-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2dcd5d52-978a-439f-9aa4-a243bc497147" (UID: "2dcd5d52-978a-439f-9aa4-a243bc497147"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:58:03.943554 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:03.943436 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dcd5d52-978a-439f-9aa4-a243bc497147-istio-data" (OuterVolumeSpecName: "istio-data") pod "2dcd5d52-978a-439f-9aa4-a243bc497147" (UID: "2dcd5d52-978a-439f-9aa4-a243bc497147"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:58:03.943745 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:03.943621 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dcd5d52-978a-439f-9aa4-a243bc497147-workload-certs\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:58:03.943745 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:03.943642 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dcd5d52-978a-439f-9aa4-a243bc497147-istio-data\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:58:03.943745 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:03.943656 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dcd5d52-978a-439f-9aa4-a243bc497147-workload-socket\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:58:03.943745 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:03.943669 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dcd5d52-978a-439f-9aa4-a243bc497147-credential-socket\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:58:03.943745 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:03.943694 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dcd5d52-978a-439f-9aa4-a243bc497147-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2dcd5d52-978a-439f-9aa4-a243bc497147" (UID: "2dcd5d52-978a-439f-9aa4-a243bc497147"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 10:58:03.945305 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:03.945277 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2dcd5d52-978a-439f-9aa4-a243bc497147-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2dcd5d52-978a-439f-9aa4-a243bc497147" (UID: "2dcd5d52-978a-439f-9aa4-a243bc497147"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 10:58:03.945420 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:03.945369 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dcd5d52-978a-439f-9aa4-a243bc497147-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2dcd5d52-978a-439f-9aa4-a243bc497147" (UID: "2dcd5d52-978a-439f-9aa4-a243bc497147"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:58:03.945724 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:03.945703 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dcd5d52-978a-439f-9aa4-a243bc497147-kube-api-access-nspx9" (OuterVolumeSpecName: "kube-api-access-nspx9") pod "2dcd5d52-978a-439f-9aa4-a243bc497147" (UID: "2dcd5d52-978a-439f-9aa4-a243bc497147"). InnerVolumeSpecName "kube-api-access-nspx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:58:03.945835 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:03.945722 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dcd5d52-978a-439f-9aa4-a243bc497147-istio-token" (OuterVolumeSpecName: "istio-token") pod "2dcd5d52-978a-439f-9aa4-a243bc497147" (UID: "2dcd5d52-978a-439f-9aa4-a243bc497147"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:58:04.044325 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:04.044286 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dcd5d52-978a-439f-9aa4-a243bc497147-istio-podinfo\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:58:04.044325 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:04.044317 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dcd5d52-978a-439f-9aa4-a243bc497147-istio-token\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:58:04.044325 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:04.044331 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dcd5d52-978a-439f-9aa4-a243bc497147-istiod-ca-cert\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:58:04.044598 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:04.044342 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dcd5d52-978a-439f-9aa4-a243bc497147-istio-envoy\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:58:04.044598 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:04.044351 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nspx9\" (UniqueName: \"kubernetes.io/projected/2dcd5d52-978a-439f-9aa4-a243bc497147-kube-api-access-nspx9\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:58:04.267492 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:04.267395 2580 generic.go:356] "Generic (PLEG): container finished" podID="2dcd5d52-978a-439f-9aa4-a243bc497147" containerID="a656586df1456762457d9b0ab6b49120cf2086b16cdc07816c05b05226409aa9" exitCode=0 Aug 04 10:58:04.267492 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:04.267454 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" Aug 04 10:58:04.267492 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:04.267473 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" event={"ID":"2dcd5d52-978a-439f-9aa4-a243bc497147","Type":"ContainerDied","Data":"a656586df1456762457d9b0ab6b49120cf2086b16cdc07816c05b05226409aa9"} Aug 04 10:58:04.267736 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:04.267534 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q" event={"ID":"2dcd5d52-978a-439f-9aa4-a243bc497147","Type":"ContainerDied","Data":"93497443bae5f873de0bb2a356e3992e13b09da4cf18824ab7cacd939aede79b"} Aug 04 10:58:04.267736 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:04.267574 2580 scope.go:117] "RemoveContainer" containerID="a656586df1456762457d9b0ab6b49120cf2086b16cdc07816c05b05226409aa9" Aug 04 10:58:04.276879 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:04.276858 2580 scope.go:117] "RemoveContainer" containerID="c5908a46e81d23bf4fdfd73faea55edf84bc6e7c6212de5d480bc5a229c47bab" Aug 04 10:58:04.287201 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:04.287181 2580 scope.go:117] "RemoveContainer" containerID="a656586df1456762457d9b0ab6b49120cf2086b16cdc07816c05b05226409aa9" Aug 04 10:58:04.287489 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:58:04.287466 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a656586df1456762457d9b0ab6b49120cf2086b16cdc07816c05b05226409aa9\": container with ID starting with a656586df1456762457d9b0ab6b49120cf2086b16cdc07816c05b05226409aa9 not found: ID does not exist" containerID="a656586df1456762457d9b0ab6b49120cf2086b16cdc07816c05b05226409aa9" Aug 04 10:58:04.287528 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:04.287505 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a656586df1456762457d9b0ab6b49120cf2086b16cdc07816c05b05226409aa9"} err="failed to get container status \"a656586df1456762457d9b0ab6b49120cf2086b16cdc07816c05b05226409aa9\": rpc error: code = NotFound desc = could not find container \"a656586df1456762457d9b0ab6b49120cf2086b16cdc07816c05b05226409aa9\": container with ID starting with a656586df1456762457d9b0ab6b49120cf2086b16cdc07816c05b05226409aa9 not found: ID does not exist" Aug 04 10:58:04.287587 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:04.287532 2580 scope.go:117] "RemoveContainer" containerID="c5908a46e81d23bf4fdfd73faea55edf84bc6e7c6212de5d480bc5a229c47bab" Aug 04 10:58:04.287827 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:58:04.287807 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5908a46e81d23bf4fdfd73faea55edf84bc6e7c6212de5d480bc5a229c47bab\": container with ID starting with c5908a46e81d23bf4fdfd73faea55edf84bc6e7c6212de5d480bc5a229c47bab not found: ID does not exist" containerID="c5908a46e81d23bf4fdfd73faea55edf84bc6e7c6212de5d480bc5a229c47bab" Aug 04 10:58:04.287895 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:04.287835 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5908a46e81d23bf4fdfd73faea55edf84bc6e7c6212de5d480bc5a229c47bab"} err="failed to get container status \"c5908a46e81d23bf4fdfd73faea55edf84bc6e7c6212de5d480bc5a229c47bab\": rpc error: code = NotFound desc = could not find container \"c5908a46e81d23bf4fdfd73faea55edf84bc6e7c6212de5d480bc5a229c47bab\": container with ID starting with c5908a46e81d23bf4fdfd73faea55edf84bc6e7c6212de5d480bc5a229c47bab not found: ID does not exist" Aug 04 10:58:04.292087 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:04.292062 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q"] Aug 04 10:58:04.295465 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:04.295439 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-03e71fbe-gw-openshift-default-78c4c48d56-kjp9q"] Aug 04 10:58:05.099475 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:05.099440 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dcd5d52-978a-439f-9aa4-a243bc497147" path="/var/lib/kubelet/pods/2dcd5d52-978a-439f-9aa4-a243bc497147/volumes" Aug 04 10:58:10.938329 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:10.938291 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:58:16.340609 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:16.340542 2580 generic.go:356] "Generic (PLEG): container finished" podID="a4246b32-957a-4473-b784-5fc5174e24e2" containerID="5dc5ea4157ba4c2e6ec55ade6d767a9ed0d7be340beb9e4f7cc02a0496741537" exitCode=0 Aug 04 10:58:16.341037 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:16.340623 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" event={"ID":"a4246b32-957a-4473-b784-5fc5174e24e2","Type":"ContainerDied","Data":"5dc5ea4157ba4c2e6ec55ade6d767a9ed0d7be340beb9e4f7cc02a0496741537"} Aug 04 10:58:16.341037 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:16.340999 2580 scope.go:117] "RemoveContainer" containerID="5dc5ea4157ba4c2e6ec55ade6d767a9ed0d7be340beb9e4f7cc02a0496741537" Aug 04 10:58:16.342034 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:16.342007 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:58:16.342092 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:16.342077 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:58:16.342134 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:16.342121 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:58:17.256443 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:17.256409 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:58:17.347691 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:17.347658 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" event={"ID":"a4246b32-957a-4473-b784-5fc5174e24e2","Type":"ContainerStarted","Data":"32975b5782377f62b79165a2816eb42aef4ed99476953a28ab77c3893f5139cc"} Aug 04 10:58:17.823336 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:17.823291 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:58:17.824499 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:17.824470 2580 patch_prober.go:28] interesting pod/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Aug 04 10:58:17.824591 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:17.824529 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" podUID="a4246b32-957a-4473-b784-5fc5174e24e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Aug 04 10:58:18.823486 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:18.823439 2580 patch_prober.go:28] interesting pod/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Aug 04 10:58:18.823910 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:18.823503 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" podUID="a4246b32-957a-4473-b784-5fc5174e24e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Aug 04 10:58:19.823674 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:19.823631 2580 patch_prober.go:28] interesting pod/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Aug 04 10:58:19.824071 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:19.823694 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" podUID="a4246b32-957a-4473-b784-5fc5174e24e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Aug 04 10:58:20.823753 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:20.823714 2580 patch_prober.go:28] interesting pod/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Aug 04 10:58:20.824164 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:20.823772 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" podUID="a4246b32-957a-4473-b784-5fc5174e24e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Aug 04 10:58:21.823541 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:21.823502 2580 patch_prober.go:28] interesting pod/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Aug 04 10:58:21.823765 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:21.823602 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" podUID="a4246b32-957a-4473-b784-5fc5174e24e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Aug 04 10:58:22.826739 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:22.826709 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:58:22.827134 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:22.827077 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:58:22.827791 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:22.827766 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:58:44.875031 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:44.874990 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq"] Aug 04 10:58:44.875551 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:44.875274 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" podUID="a4246b32-957a-4473-b784-5fc5174e24e2" containerName="istio-proxy" containerID="cri-o://32975b5782377f62b79165a2816eb42aef4ed99476953a28ab77c3893f5139cc" gracePeriod=30 Aug 04 10:58:45.703543 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.703508 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt"] Aug 04 10:58:45.703958 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.703939 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dcd5d52-978a-439f-9aa4-a243bc497147" containerName="istio-proxy" Aug 04 10:58:45.704047 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.703959 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dcd5d52-978a-439f-9aa4-a243bc497147" containerName="istio-proxy" Aug 04 10:58:45.704102 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.704073 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dcd5d52-978a-439f-9aa4-a243bc497147" containerName="istio-proxy" Aug 04 10:58:45.704102 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.704086 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dcd5d52-978a-439f-9aa4-a243bc497147" containerName="istio-proxy" Aug 04 10:58:45.712769 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.712728 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:45.716716 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.716689 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-524f2cac-openshift-default-dockercfg-dvc25\"" Aug 04 10:58:45.721892 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.721865 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt"] Aug 04 10:58:45.829843 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.829807 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f694f65a-c102-47ed-8a3d-943448d04892-credential-socket\") pod \"e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:45.829843 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.829841 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f694f65a-c102-47ed-8a3d-943448d04892-istio-data\") pod \"e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:45.830079 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.829859 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f694f65a-c102-47ed-8a3d-943448d04892-istio-token\") pod \"e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:45.830079 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.829878 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f694f65a-c102-47ed-8a3d-943448d04892-istio-envoy\") pod \"e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:45.830079 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.829935 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ldfw\" (UniqueName: \"kubernetes.io/projected/f694f65a-c102-47ed-8a3d-943448d04892-kube-api-access-9ldfw\") pod \"e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:45.830079 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.829991 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f694f65a-c102-47ed-8a3d-943448d04892-istio-podinfo\") pod \"e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:45.830079 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.830017 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f694f65a-c102-47ed-8a3d-943448d04892-istiod-ca-cert\") pod \"e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:45.830307 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.830115 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f694f65a-c102-47ed-8a3d-943448d04892-workload-certs\") pod \"e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:45.830307 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.830209 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f694f65a-c102-47ed-8a3d-943448d04892-workload-socket\") pod \"e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:45.931393 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.931350 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f694f65a-c102-47ed-8a3d-943448d04892-workload-socket\") pod \"e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:45.931393 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.931395 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f694f65a-c102-47ed-8a3d-943448d04892-credential-socket\") pod \"e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:45.931953 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.931527 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f694f65a-c102-47ed-8a3d-943448d04892-istio-data\") pod \"e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:45.931953 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.931596 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f694f65a-c102-47ed-8a3d-943448d04892-istio-token\") pod \"e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:45.931953 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.931628 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f694f65a-c102-47ed-8a3d-943448d04892-istio-envoy\") pod \"e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:45.931953 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.931648 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ldfw\" (UniqueName: \"kubernetes.io/projected/f694f65a-c102-47ed-8a3d-943448d04892-kube-api-access-9ldfw\") pod \"e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:45.931953 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.931689 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f694f65a-c102-47ed-8a3d-943448d04892-istio-podinfo\") pod \"e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:45.931953 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.931716 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f694f65a-c102-47ed-8a3d-943448d04892-istiod-ca-cert\") pod \"e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:45.931953 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.931799 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f694f65a-c102-47ed-8a3d-943448d04892-workload-certs\") pod \"e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:45.931953 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.931853 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f694f65a-c102-47ed-8a3d-943448d04892-credential-socket\") pod \"e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:45.932326 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.932028 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f694f65a-c102-47ed-8a3d-943448d04892-istio-data\") pod \"e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:45.932326 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.932057 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f694f65a-c102-47ed-8a3d-943448d04892-workload-socket\") pod \"e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:45.932326 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.932216 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f694f65a-c102-47ed-8a3d-943448d04892-workload-certs\") pod \"e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:45.932480 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.932459 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f694f65a-c102-47ed-8a3d-943448d04892-istiod-ca-cert\") pod \"e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:45.934005 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.933986 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f694f65a-c102-47ed-8a3d-943448d04892-istio-envoy\") pod \"e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:45.934294 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.934272 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f694f65a-c102-47ed-8a3d-943448d04892-istio-podinfo\") pod \"e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:45.941498 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.941476 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f694f65a-c102-47ed-8a3d-943448d04892-istio-token\") pod \"e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:45.941984 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:45.941963 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ldfw\" (UniqueName: \"kubernetes.io/projected/f694f65a-c102-47ed-8a3d-943448d04892-kube-api-access-9ldfw\") pod \"e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:46.025316 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:46.025228 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:46.170428 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:46.170385 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt"] Aug 04 10:58:46.173623 ip-10-0-142-113 kubenswrapper[2580]: W0804 10:58:46.173593 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf694f65a_c102_47ed_8a3d_943448d04892.slice/crio-1c29103062d694037dd735f7ed41801e4b2dd9ee66e28efcf79e3b5e43174f1c WatchSource:0}: Error finding container 1c29103062d694037dd735f7ed41801e4b2dd9ee66e28efcf79e3b5e43174f1c: Status 404 returned error can't find the container with id 1c29103062d694037dd735f7ed41801e4b2dd9ee66e28efcf79e3b5e43174f1c Aug 04 10:58:46.175980 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:46.175939 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:58:46.176071 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:46.176029 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:58:46.176111 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:46.176072 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:58:46.487374 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:46.487338 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" event={"ID":"f694f65a-c102-47ed-8a3d-943448d04892","Type":"ContainerStarted","Data":"750732e77c631128298bc3de0448f398ea142eb5710e5660b35aaf8672502804"} Aug 04 10:58:46.487612 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:46.487397 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" event={"ID":"f694f65a-c102-47ed-8a3d-943448d04892","Type":"ContainerStarted","Data":"1c29103062d694037dd735f7ed41801e4b2dd9ee66e28efcf79e3b5e43174f1c"} Aug 04 10:58:46.509896 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:46.509842 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" podStartSLOduration=1.509824182 podStartE2EDuration="1.509824182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:58:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:58:46.509183935 +0000 UTC m=+1865.930071347" watchObservedRunningTime="2026-08-04 10:58:46.509824182 +0000 UTC m=+1865.930711629" Aug 04 10:58:46.536430 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:46.536394 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:58:47.026267 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:47.026229 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:47.031338 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:47.031313 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:47.492059 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:47.492029 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:47.492974 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:47.492953 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:58:50.138270 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.138244 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:58:50.270810 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.270728 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5bbp\" (UniqueName: \"kubernetes.io/projected/a4246b32-957a-4473-b784-5fc5174e24e2-kube-api-access-w5bbp\") pod \"a4246b32-957a-4473-b784-5fc5174e24e2\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " Aug 04 10:58:50.270810 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.270775 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4246b32-957a-4473-b784-5fc5174e24e2-istio-envoy\") pod \"a4246b32-957a-4473-b784-5fc5174e24e2\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " Aug 04 10:58:50.270996 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.270837 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4246b32-957a-4473-b784-5fc5174e24e2-istio-data\") pod \"a4246b32-957a-4473-b784-5fc5174e24e2\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " Aug 04 10:58:50.270996 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.270894 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4246b32-957a-4473-b784-5fc5174e24e2-istiod-ca-cert\") pod \"a4246b32-957a-4473-b784-5fc5174e24e2\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " Aug 04 10:58:50.270996 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.270923 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4246b32-957a-4473-b784-5fc5174e24e2-istio-podinfo\") pod \"a4246b32-957a-4473-b784-5fc5174e24e2\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " Aug 04 10:58:50.270996 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.270953 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4246b32-957a-4473-b784-5fc5174e24e2-workload-socket\") pod \"a4246b32-957a-4473-b784-5fc5174e24e2\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " Aug 04 10:58:50.271201 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.271000 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4246b32-957a-4473-b784-5fc5174e24e2-workload-certs\") pod \"a4246b32-957a-4473-b784-5fc5174e24e2\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " Aug 04 10:58:50.271201 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.271031 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4246b32-957a-4473-b784-5fc5174e24e2-istio-token\") pod \"a4246b32-957a-4473-b784-5fc5174e24e2\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " Aug 04 10:58:50.271201 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.271096 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4246b32-957a-4473-b784-5fc5174e24e2-credential-socket\") pod \"a4246b32-957a-4473-b784-5fc5174e24e2\" (UID: \"a4246b32-957a-4473-b784-5fc5174e24e2\") " Aug 04 10:58:50.271350 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.271191 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4246b32-957a-4473-b784-5fc5174e24e2-istio-data" (OuterVolumeSpecName: "istio-data") pod "a4246b32-957a-4473-b784-5fc5174e24e2" (UID: "a4246b32-957a-4473-b784-5fc5174e24e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:58:50.271415 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.271343 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4246b32-957a-4473-b784-5fc5174e24e2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a4246b32-957a-4473-b784-5fc5174e24e2" (UID: "a4246b32-957a-4473-b784-5fc5174e24e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:58:50.271415 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.271361 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4246b32-957a-4473-b784-5fc5174e24e2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a4246b32-957a-4473-b784-5fc5174e24e2" (UID: "a4246b32-957a-4473-b784-5fc5174e24e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 10:58:50.271415 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.271399 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4246b32-957a-4473-b784-5fc5174e24e2-istio-data\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:58:50.271622 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.271600 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4246b32-957a-4473-b784-5fc5174e24e2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a4246b32-957a-4473-b784-5fc5174e24e2" (UID: "a4246b32-957a-4473-b784-5fc5174e24e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:58:50.271854 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.271828 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4246b32-957a-4473-b784-5fc5174e24e2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a4246b32-957a-4473-b784-5fc5174e24e2" (UID: "a4246b32-957a-4473-b784-5fc5174e24e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:58:50.273177 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.273143 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4246b32-957a-4473-b784-5fc5174e24e2-kube-api-access-w5bbp" (OuterVolumeSpecName: "kube-api-access-w5bbp") pod "a4246b32-957a-4473-b784-5fc5174e24e2" (UID: "a4246b32-957a-4473-b784-5fc5174e24e2"). InnerVolumeSpecName "kube-api-access-w5bbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:58:50.273462 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.273434 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4246b32-957a-4473-b784-5fc5174e24e2-istio-token" (OuterVolumeSpecName: "istio-token") pod "a4246b32-957a-4473-b784-5fc5174e24e2" (UID: "a4246b32-957a-4473-b784-5fc5174e24e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:58:50.273617 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.273590 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4246b32-957a-4473-b784-5fc5174e24e2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a4246b32-957a-4473-b784-5fc5174e24e2" (UID: "a4246b32-957a-4473-b784-5fc5174e24e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:58:50.273879 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.273805 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4246b32-957a-4473-b784-5fc5174e24e2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a4246b32-957a-4473-b784-5fc5174e24e2" (UID: "a4246b32-957a-4473-b784-5fc5174e24e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 10:58:50.372707 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.372670 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5bbp\" (UniqueName: \"kubernetes.io/projected/a4246b32-957a-4473-b784-5fc5174e24e2-kube-api-access-w5bbp\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:58:50.372707 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.372701 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4246b32-957a-4473-b784-5fc5174e24e2-istio-envoy\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:58:50.372707 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.372710 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4246b32-957a-4473-b784-5fc5174e24e2-istiod-ca-cert\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:58:50.372707 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.372718 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4246b32-957a-4473-b784-5fc5174e24e2-istio-podinfo\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:58:50.372976 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.372726 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4246b32-957a-4473-b784-5fc5174e24e2-workload-socket\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:58:50.372976 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.372735 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4246b32-957a-4473-b784-5fc5174e24e2-workload-certs\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:58:50.372976 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.372742 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4246b32-957a-4473-b784-5fc5174e24e2-istio-token\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:58:50.372976 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.372750 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4246b32-957a-4473-b784-5fc5174e24e2-credential-socket\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:58:50.507215 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.507180 2580 generic.go:356] "Generic (PLEG): container finished" podID="a4246b32-957a-4473-b784-5fc5174e24e2" containerID="32975b5782377f62b79165a2816eb42aef4ed99476953a28ab77c3893f5139cc" exitCode=0 Aug 04 10:58:50.507417 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.507230 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" Aug 04 10:58:50.507417 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.507270 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" event={"ID":"a4246b32-957a-4473-b784-5fc5174e24e2","Type":"ContainerDied","Data":"32975b5782377f62b79165a2816eb42aef4ed99476953a28ab77c3893f5139cc"} Aug 04 10:58:50.507417 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.507339 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq" event={"ID":"a4246b32-957a-4473-b784-5fc5174e24e2","Type":"ContainerDied","Data":"a7674e72ec4b5151c2a8181050f04b100fea35afbdca9892be6ea8f4067e9da5"} Aug 04 10:58:50.507417 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.507370 2580 scope.go:117] "RemoveContainer" containerID="32975b5782377f62b79165a2816eb42aef4ed99476953a28ab77c3893f5139cc" Aug 04 10:58:50.516627 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.516609 2580 scope.go:117] "RemoveContainer" containerID="5dc5ea4157ba4c2e6ec55ade6d767a9ed0d7be340beb9e4f7cc02a0496741537" Aug 04 10:58:50.524601 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.524538 2580 scope.go:117] "RemoveContainer" containerID="32975b5782377f62b79165a2816eb42aef4ed99476953a28ab77c3893f5139cc" Aug 04 10:58:50.524899 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:58:50.524872 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32975b5782377f62b79165a2816eb42aef4ed99476953a28ab77c3893f5139cc\": container with ID starting with 32975b5782377f62b79165a2816eb42aef4ed99476953a28ab77c3893f5139cc not found: ID does not exist" containerID="32975b5782377f62b79165a2816eb42aef4ed99476953a28ab77c3893f5139cc" Aug 04 10:58:50.524946 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.524905 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32975b5782377f62b79165a2816eb42aef4ed99476953a28ab77c3893f5139cc"} err="failed to get container status \"32975b5782377f62b79165a2816eb42aef4ed99476953a28ab77c3893f5139cc\": rpc error: code = NotFound desc = could not find container \"32975b5782377f62b79165a2816eb42aef4ed99476953a28ab77c3893f5139cc\": container with ID starting with 32975b5782377f62b79165a2816eb42aef4ed99476953a28ab77c3893f5139cc not found: ID does not exist" Aug 04 10:58:50.524946 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.524928 2580 scope.go:117] "RemoveContainer" containerID="5dc5ea4157ba4c2e6ec55ade6d767a9ed0d7be340beb9e4f7cc02a0496741537" Aug 04 10:58:50.525185 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:58:50.525164 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dc5ea4157ba4c2e6ec55ade6d767a9ed0d7be340beb9e4f7cc02a0496741537\": container with ID starting with 5dc5ea4157ba4c2e6ec55ade6d767a9ed0d7be340beb9e4f7cc02a0496741537 not found: ID does not exist" containerID="5dc5ea4157ba4c2e6ec55ade6d767a9ed0d7be340beb9e4f7cc02a0496741537" Aug 04 10:58:50.525284 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.525189 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dc5ea4157ba4c2e6ec55ade6d767a9ed0d7be340beb9e4f7cc02a0496741537"} err="failed to get container status \"5dc5ea4157ba4c2e6ec55ade6d767a9ed0d7be340beb9e4f7cc02a0496741537\": rpc error: code = NotFound desc = could not find container \"5dc5ea4157ba4c2e6ec55ade6d767a9ed0d7be340beb9e4f7cc02a0496741537\": container with ID starting with 5dc5ea4157ba4c2e6ec55ade6d767a9ed0d7be340beb9e4f7cc02a0496741537 not found: ID does not exist" Aug 04 10:58:50.529620 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.529597 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq"] Aug 04 10:58:50.535509 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:50.535488 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-fb58dd84-openshift-default-65bbdcc74-sncrq"] Aug 04 10:58:51.099135 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:51.099099 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4246b32-957a-4473-b784-5fc5174e24e2" path="/var/lib/kubelet/pods/a4246b32-957a-4473-b784-5fc5174e24e2/volumes" Aug 04 10:58:56.048495 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:58:56.048462 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:59:01.562873 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:01.562779 2580 generic.go:356] "Generic (PLEG): container finished" podID="f694f65a-c102-47ed-8a3d-943448d04892" containerID="750732e77c631128298bc3de0448f398ea142eb5710e5660b35aaf8672502804" exitCode=0 Aug 04 10:59:01.562873 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:01.562853 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" event={"ID":"f694f65a-c102-47ed-8a3d-943448d04892","Type":"ContainerDied","Data":"750732e77c631128298bc3de0448f398ea142eb5710e5660b35aaf8672502804"} Aug 04 10:59:01.563308 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:01.563281 2580 scope.go:117] "RemoveContainer" containerID="750732e77c631128298bc3de0448f398ea142eb5710e5660b35aaf8672502804" Aug 04 10:59:01.564352 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:01.564321 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:59:01.564427 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:01.564413 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:59:01.564472 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:01.564449 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 10:59:02.492812 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:02.492750 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:59:02.569430 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:02.569382 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" event={"ID":"f694f65a-c102-47ed-8a3d-943448d04892","Type":"ContainerStarted","Data":"fb4b30d9ed97669311b769fe8da95beb86f874d1f9b449ba0838b914f45550d2"} Aug 04 10:59:03.025827 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:03.025791 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:59:03.027072 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:03.027046 2580 patch_prober.go:28] interesting pod/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Aug 04 10:59:03.027181 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:03.027095 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" podUID="f694f65a-c102-47ed-8a3d-943448d04892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Aug 04 10:59:04.025884 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:04.025843 2580 patch_prober.go:28] interesting pod/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Aug 04 10:59:04.026270 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:04.025904 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" podUID="f694f65a-c102-47ed-8a3d-943448d04892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Aug 04 10:59:05.026015 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:05.025976 2580 patch_prober.go:28] interesting pod/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Aug 04 10:59:05.026394 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:05.026037 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" podUID="f694f65a-c102-47ed-8a3d-943448d04892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Aug 04 10:59:06.025804 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:06.025764 2580 patch_prober.go:28] interesting pod/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Aug 04 10:59:06.026009 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:06.025823 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" podUID="f694f65a-c102-47ed-8a3d-943448d04892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Aug 04 10:59:07.026094 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:07.026056 2580 patch_prober.go:28] interesting pod/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Aug 04 10:59:07.026484 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:07.026114 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" podUID="f694f65a-c102-47ed-8a3d-943448d04892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Aug 04 10:59:08.030141 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:08.030106 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:59:08.030636 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:08.030505 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:59:08.031107 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:08.031088 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:59:19.900353 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:19.900292 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt"] Aug 04 10:59:19.901326 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:19.901266 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" podUID="f694f65a-c102-47ed-8a3d-943448d04892" containerName="istio-proxy" containerID="cri-o://fb4b30d9ed97669311b769fe8da95beb86f874d1f9b449ba0838b914f45550d2" gracePeriod=30 Aug 04 10:59:22.538195 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:22.538153 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 10:59:23.032775 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:23.032741 2580 patch_prober.go:28] interesting pod/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Aug 04 10:59:23.032946 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:23.032810 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" podUID="f694f65a-c102-47ed-8a3d-943448d04892" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 10:59:25.159002 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.158967 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:59:25.294375 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.294279 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f694f65a-c102-47ed-8a3d-943448d04892-istio-data\") pod \"f694f65a-c102-47ed-8a3d-943448d04892\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " Aug 04 10:59:25.294375 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.294336 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f694f65a-c102-47ed-8a3d-943448d04892-credential-socket\") pod \"f694f65a-c102-47ed-8a3d-943448d04892\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " Aug 04 10:59:25.294657 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.294387 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f694f65a-c102-47ed-8a3d-943448d04892-istio-envoy\") pod \"f694f65a-c102-47ed-8a3d-943448d04892\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " Aug 04 10:59:25.294657 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.294414 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f694f65a-c102-47ed-8a3d-943448d04892-istio-token\") pod \"f694f65a-c102-47ed-8a3d-943448d04892\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " Aug 04 10:59:25.294657 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.294438 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f694f65a-c102-47ed-8a3d-943448d04892-istiod-ca-cert\") pod \"f694f65a-c102-47ed-8a3d-943448d04892\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " Aug 04 10:59:25.294657 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.294481 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f694f65a-c102-47ed-8a3d-943448d04892-workload-socket\") pod \"f694f65a-c102-47ed-8a3d-943448d04892\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " Aug 04 10:59:25.294657 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.294524 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f694f65a-c102-47ed-8a3d-943448d04892-istio-podinfo\") pod \"f694f65a-c102-47ed-8a3d-943448d04892\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " Aug 04 10:59:25.294657 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.294595 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f694f65a-c102-47ed-8a3d-943448d04892-workload-certs\") pod \"f694f65a-c102-47ed-8a3d-943448d04892\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " Aug 04 10:59:25.294657 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.294637 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ldfw\" (UniqueName: \"kubernetes.io/projected/f694f65a-c102-47ed-8a3d-943448d04892-kube-api-access-9ldfw\") pod \"f694f65a-c102-47ed-8a3d-943448d04892\" (UID: \"f694f65a-c102-47ed-8a3d-943448d04892\") " Aug 04 10:59:25.294657 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.294638 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f694f65a-c102-47ed-8a3d-943448d04892-istio-data" (OuterVolumeSpecName: "istio-data") pod "f694f65a-c102-47ed-8a3d-943448d04892" (UID: "f694f65a-c102-47ed-8a3d-943448d04892"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:59:25.294657 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.294660 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f694f65a-c102-47ed-8a3d-943448d04892-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f694f65a-c102-47ed-8a3d-943448d04892" (UID: "f694f65a-c102-47ed-8a3d-943448d04892"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:59:25.295115 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.294869 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f694f65a-c102-47ed-8a3d-943448d04892-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f694f65a-c102-47ed-8a3d-943448d04892" (UID: "f694f65a-c102-47ed-8a3d-943448d04892"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:59:25.295115 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.294861 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f694f65a-c102-47ed-8a3d-943448d04892-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f694f65a-c102-47ed-8a3d-943448d04892" (UID: "f694f65a-c102-47ed-8a3d-943448d04892"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:59:25.295115 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.294913 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f694f65a-c102-47ed-8a3d-943448d04892-istio-data\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:59:25.295115 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.294932 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f694f65a-c102-47ed-8a3d-943448d04892-credential-socket\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:59:25.295115 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.294938 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f694f65a-c102-47ed-8a3d-943448d04892-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f694f65a-c102-47ed-8a3d-943448d04892" (UID: "f694f65a-c102-47ed-8a3d-943448d04892"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 10:59:25.296966 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.296939 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f694f65a-c102-47ed-8a3d-943448d04892-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f694f65a-c102-47ed-8a3d-943448d04892" (UID: "f694f65a-c102-47ed-8a3d-943448d04892"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:59:25.296966 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.296946 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f694f65a-c102-47ed-8a3d-943448d04892-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f694f65a-c102-47ed-8a3d-943448d04892" (UID: "f694f65a-c102-47ed-8a3d-943448d04892"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 10:59:25.297099 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.296983 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f694f65a-c102-47ed-8a3d-943448d04892-kube-api-access-9ldfw" (OuterVolumeSpecName: "kube-api-access-9ldfw") pod "f694f65a-c102-47ed-8a3d-943448d04892" (UID: "f694f65a-c102-47ed-8a3d-943448d04892"). InnerVolumeSpecName "kube-api-access-9ldfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:59:25.297099 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.297077 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f694f65a-c102-47ed-8a3d-943448d04892-istio-token" (OuterVolumeSpecName: "istio-token") pod "f694f65a-c102-47ed-8a3d-943448d04892" (UID: "f694f65a-c102-47ed-8a3d-943448d04892"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:59:25.395893 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.395852 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ldfw\" (UniqueName: \"kubernetes.io/projected/f694f65a-c102-47ed-8a3d-943448d04892-kube-api-access-9ldfw\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:59:25.395893 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.395884 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f694f65a-c102-47ed-8a3d-943448d04892-istio-envoy\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:59:25.395893 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.395893 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f694f65a-c102-47ed-8a3d-943448d04892-istio-token\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:59:25.395893 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.395900 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f694f65a-c102-47ed-8a3d-943448d04892-istiod-ca-cert\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:59:25.396161 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.395909 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f694f65a-c102-47ed-8a3d-943448d04892-workload-socket\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:59:25.396161 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.395917 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f694f65a-c102-47ed-8a3d-943448d04892-istio-podinfo\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:59:25.396161 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.395925 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f694f65a-c102-47ed-8a3d-943448d04892-workload-certs\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 10:59:25.693594 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.693541 2580 generic.go:356] "Generic (PLEG): container finished" podID="f694f65a-c102-47ed-8a3d-943448d04892" containerID="fb4b30d9ed97669311b769fe8da95beb86f874d1f9b449ba0838b914f45550d2" exitCode=0 Aug 04 10:59:25.693788 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.693618 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" event={"ID":"f694f65a-c102-47ed-8a3d-943448d04892","Type":"ContainerDied","Data":"fb4b30d9ed97669311b769fe8da95beb86f874d1f9b449ba0838b914f45550d2"} Aug 04 10:59:25.693788 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.693635 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" Aug 04 10:59:25.693788 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.693694 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt" event={"ID":"f694f65a-c102-47ed-8a3d-943448d04892","Type":"ContainerDied","Data":"1c29103062d694037dd735f7ed41801e4b2dd9ee66e28efcf79e3b5e43174f1c"} Aug 04 10:59:25.693788 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.693715 2580 scope.go:117] "RemoveContainer" containerID="fb4b30d9ed97669311b769fe8da95beb86f874d1f9b449ba0838b914f45550d2" Aug 04 10:59:25.703715 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.703695 2580 scope.go:117] "RemoveContainer" containerID="750732e77c631128298bc3de0448f398ea142eb5710e5660b35aaf8672502804" Aug 04 10:59:25.711834 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.711816 2580 scope.go:117] "RemoveContainer" containerID="fb4b30d9ed97669311b769fe8da95beb86f874d1f9b449ba0838b914f45550d2" Aug 04 10:59:25.712099 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:59:25.712079 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb4b30d9ed97669311b769fe8da95beb86f874d1f9b449ba0838b914f45550d2\": container with ID starting with fb4b30d9ed97669311b769fe8da95beb86f874d1f9b449ba0838b914f45550d2 not found: ID does not exist" containerID="fb4b30d9ed97669311b769fe8da95beb86f874d1f9b449ba0838b914f45550d2" Aug 04 10:59:25.712172 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.712112 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4b30d9ed97669311b769fe8da95beb86f874d1f9b449ba0838b914f45550d2"} err="failed to get container status \"fb4b30d9ed97669311b769fe8da95beb86f874d1f9b449ba0838b914f45550d2\": rpc error: code = NotFound desc = could not find container \"fb4b30d9ed97669311b769fe8da95beb86f874d1f9b449ba0838b914f45550d2\": container with ID starting with fb4b30d9ed97669311b769fe8da95beb86f874d1f9b449ba0838b914f45550d2 not found: ID does not exist" Aug 04 10:59:25.712172 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.712139 2580 scope.go:117] "RemoveContainer" containerID="750732e77c631128298bc3de0448f398ea142eb5710e5660b35aaf8672502804" Aug 04 10:59:25.712412 ip-10-0-142-113 kubenswrapper[2580]: E0804 10:59:25.712397 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"750732e77c631128298bc3de0448f398ea142eb5710e5660b35aaf8672502804\": container with ID starting with 750732e77c631128298bc3de0448f398ea142eb5710e5660b35aaf8672502804 not found: ID does not exist" containerID="750732e77c631128298bc3de0448f398ea142eb5710e5660b35aaf8672502804" Aug 04 10:59:25.712468 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.712419 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"750732e77c631128298bc3de0448f398ea142eb5710e5660b35aaf8672502804"} err="failed to get container status \"750732e77c631128298bc3de0448f398ea142eb5710e5660b35aaf8672502804\": rpc error: code = NotFound desc = could not find container \"750732e77c631128298bc3de0448f398ea142eb5710e5660b35aaf8672502804\": container with ID starting with 750732e77c631128298bc3de0448f398ea142eb5710e5660b35aaf8672502804 not found: ID does not exist" Aug 04 10:59:25.716306 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.716275 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt"] Aug 04 10:59:25.722528 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:25.722503 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-524f2cac-openshift-default-8dc7d7968-qwbwt"] Aug 04 10:59:27.099086 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:27.099049 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f694f65a-c102-47ed-8a3d-943448d04892" path="/var/lib/kubelet/pods/f694f65a-c102-47ed-8a3d-943448d04892/volumes" Aug 04 10:59:35.530283 ip-10-0-142-113 kubenswrapper[2580]: I0804 10:59:35.530233 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:01:11.394502 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:11.394467 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:01:18.441203 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:18.441167 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:01:22.339006 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.338971 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw"] Aug 04 11:01:22.339373 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.339329 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4246b32-957a-4473-b784-5fc5174e24e2" containerName="istio-proxy" Aug 04 11:01:22.339373 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.339343 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4246b32-957a-4473-b784-5fc5174e24e2" containerName="istio-proxy" Aug 04 11:01:22.339373 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.339365 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dcd5d52-978a-439f-9aa4-a243bc497147" containerName="istio-proxy" Aug 04 11:01:22.339373 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.339371 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dcd5d52-978a-439f-9aa4-a243bc497147" containerName="istio-proxy" Aug 04 11:01:22.339548 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.339381 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f694f65a-c102-47ed-8a3d-943448d04892" containerName="istio-proxy" Aug 04 11:01:22.339548 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.339387 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f694f65a-c102-47ed-8a3d-943448d04892" containerName="istio-proxy" Aug 04 11:01:22.339548 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.339394 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f694f65a-c102-47ed-8a3d-943448d04892" containerName="istio-proxy" Aug 04 11:01:22.339548 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.339399 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f694f65a-c102-47ed-8a3d-943448d04892" containerName="istio-proxy" Aug 04 11:01:22.339548 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.339458 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4246b32-957a-4473-b784-5fc5174e24e2" containerName="istio-proxy" Aug 04 11:01:22.339548 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.339465 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f694f65a-c102-47ed-8a3d-943448d04892" containerName="istio-proxy" Aug 04 11:01:22.339548 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.339472 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4246b32-957a-4473-b784-5fc5174e24e2" containerName="istio-proxy" Aug 04 11:01:22.342846 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.342827 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:22.345746 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.345727 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-fe15eb98-openshift-default-dockercfg-sp7t6\"" Aug 04 11:01:22.355844 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.355816 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw"] Aug 04 11:01:22.501900 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.501864 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/469a686d-3a72-4262-b5e2-1e5357ec90a5-workload-certs\") pod \"e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:22.502102 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.501917 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/469a686d-3a72-4262-b5e2-1e5357ec90a5-workload-socket\") pod \"e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:22.502102 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.501975 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72q4m\" (UniqueName: \"kubernetes.io/projected/469a686d-3a72-4262-b5e2-1e5357ec90a5-kube-api-access-72q4m\") pod \"e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:22.502102 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.502003 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/469a686d-3a72-4262-b5e2-1e5357ec90a5-istio-podinfo\") pod \"e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:22.502102 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.502082 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/469a686d-3a72-4262-b5e2-1e5357ec90a5-istio-data\") pod \"e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:22.502314 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.502121 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/469a686d-3a72-4262-b5e2-1e5357ec90a5-credential-socket\") pod \"e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:22.502314 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.502151 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/469a686d-3a72-4262-b5e2-1e5357ec90a5-istiod-ca-cert\") pod \"e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:22.502314 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.502214 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/469a686d-3a72-4262-b5e2-1e5357ec90a5-istio-token\") pod \"e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:22.502314 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.502242 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/469a686d-3a72-4262-b5e2-1e5357ec90a5-istio-envoy\") pod \"e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:22.602740 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.602701 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/469a686d-3a72-4262-b5e2-1e5357ec90a5-istio-token\") pod \"e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:22.602931 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.602750 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/469a686d-3a72-4262-b5e2-1e5357ec90a5-istio-envoy\") pod \"e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:22.602931 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.602790 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/469a686d-3a72-4262-b5e2-1e5357ec90a5-workload-certs\") pod \"e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:22.602931 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.602822 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/469a686d-3a72-4262-b5e2-1e5357ec90a5-workload-socket\") pod \"e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:22.602931 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.602881 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72q4m\" (UniqueName: \"kubernetes.io/projected/469a686d-3a72-4262-b5e2-1e5357ec90a5-kube-api-access-72q4m\") pod \"e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:22.602931 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.602910 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/469a686d-3a72-4262-b5e2-1e5357ec90a5-istio-podinfo\") pod \"e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:22.603229 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.602945 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/469a686d-3a72-4262-b5e2-1e5357ec90a5-istio-data\") pod \"e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:22.603229 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.602978 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/469a686d-3a72-4262-b5e2-1e5357ec90a5-credential-socket\") pod \"e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:22.603229 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.603005 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/469a686d-3a72-4262-b5e2-1e5357ec90a5-istiod-ca-cert\") pod \"e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:22.603455 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.603306 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/469a686d-3a72-4262-b5e2-1e5357ec90a5-workload-certs\") pod \"e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:22.603455 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.603337 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/469a686d-3a72-4262-b5e2-1e5357ec90a5-workload-socket\") pod \"e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:22.603455 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.603377 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/469a686d-3a72-4262-b5e2-1e5357ec90a5-istio-data\") pod \"e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:22.603644 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.603535 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/469a686d-3a72-4262-b5e2-1e5357ec90a5-credential-socket\") pod \"e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:22.604109 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.604086 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/469a686d-3a72-4262-b5e2-1e5357ec90a5-istiod-ca-cert\") pod \"e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:22.605327 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.605308 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/469a686d-3a72-4262-b5e2-1e5357ec90a5-istio-envoy\") pod \"e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:22.605662 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.605641 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/469a686d-3a72-4262-b5e2-1e5357ec90a5-istio-podinfo\") pod \"e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:22.611442 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.611416 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/469a686d-3a72-4262-b5e2-1e5357ec90a5-istio-token\") pod \"e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:22.611534 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.611478 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72q4m\" (UniqueName: \"kubernetes.io/projected/469a686d-3a72-4262-b5e2-1e5357ec90a5-kube-api-access-72q4m\") pod \"e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:22.652592 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.652547 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:22.793396 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.793370 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw"] Aug 04 11:01:22.795955 ip-10-0-142-113 kubenswrapper[2580]: W0804 11:01:22.795933 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod469a686d_3a72_4262_b5e2_1e5357ec90a5.slice/crio-e6c6d955d86a60adf2f6f36a3e127d6430617f9ba9de3f4deb86a6ca3946b0b9 WatchSource:0}: Error finding container e6c6d955d86a60adf2f6f36a3e127d6430617f9ba9de3f4deb86a6ca3946b0b9: Status 404 returned error can't find the container with id e6c6d955d86a60adf2f6f36a3e127d6430617f9ba9de3f4deb86a6ca3946b0b9 Aug 04 11:01:22.798080 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.798045 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:01:22.798183 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.798111 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:01:22.798183 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:22.798141 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:01:23.249901 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:23.249810 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" event={"ID":"469a686d-3a72-4262-b5e2-1e5357ec90a5","Type":"ContainerStarted","Data":"e2562b867db89dd98d24e835fc6b663a1c3042c2c2367ad6ac53c81d7ee5cb46"} Aug 04 11:01:23.249901 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:23.249862 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" event={"ID":"469a686d-3a72-4262-b5e2-1e5357ec90a5","Type":"ContainerStarted","Data":"e6c6d955d86a60adf2f6f36a3e127d6430617f9ba9de3f4deb86a6ca3946b0b9"} Aug 04 11:01:23.274391 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:23.274339 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" podStartSLOduration=1.274324274 podStartE2EDuration="1.274324274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:01:23.273547165 +0000 UTC m=+2022.694434576" watchObservedRunningTime="2026-08-04 11:01:23.274324274 +0000 UTC m=+2022.695211747" Aug 04 11:01:23.652992 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:23.652949 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:23.657826 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:23.657804 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:24.254524 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:24.254498 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:24.255440 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:24.255419 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:30.736066 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:30.736021 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:01:40.297702 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:40.297666 2580 patch_prober.go:28] interesting pod/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Aug 04 11:01:40.298084 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:40.297731 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" podUID="469a686d-3a72-4262-b5e2-1e5357ec90a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Aug 04 11:01:41.339870 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:41.339832 2580 generic.go:356] "Generic (PLEG): container finished" podID="469a686d-3a72-4262-b5e2-1e5357ec90a5" containerID="e2562b867db89dd98d24e835fc6b663a1c3042c2c2367ad6ac53c81d7ee5cb46" exitCode=0 Aug 04 11:01:41.340352 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:41.339897 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" event={"ID":"469a686d-3a72-4262-b5e2-1e5357ec90a5","Type":"ContainerDied","Data":"e2562b867db89dd98d24e835fc6b663a1c3042c2c2367ad6ac53c81d7ee5cb46"} Aug 04 11:01:41.340352 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:41.340260 2580 scope.go:117] "RemoveContainer" containerID="e2562b867db89dd98d24e835fc6b663a1c3042c2c2367ad6ac53c81d7ee5cb46" Aug 04 11:01:41.341238 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:41.341204 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:01:41.341334 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:41.341279 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:01:41.341334 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:41.341313 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:01:42.346597 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:42.346535 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" event={"ID":"469a686d-3a72-4262-b5e2-1e5357ec90a5","Type":"ContainerStarted","Data":"3fe8c771cb58a97da4cb59fb48feab01148728be1a38876690d4f3c54148f068"} Aug 04 11:01:42.653645 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:42.653611 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:42.653861 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:42.653840 2580 patch_prober.go:28] interesting pod/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Aug 04 11:01:42.653908 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:42.653886 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" podUID="469a686d-3a72-4262-b5e2-1e5357ec90a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Aug 04 11:01:43.653379 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:43.653346 2580 patch_prober.go:28] interesting pod/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Aug 04 11:01:43.653813 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:43.653400 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" podUID="469a686d-3a72-4262-b5e2-1e5357ec90a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Aug 04 11:01:44.653795 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:44.653757 2580 patch_prober.go:28] interesting pod/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Aug 04 11:01:44.654260 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:44.653819 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" podUID="469a686d-3a72-4262-b5e2-1e5357ec90a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Aug 04 11:01:45.653982 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:45.653944 2580 patch_prober.go:28] interesting pod/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Aug 04 11:01:45.654365 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:45.654003 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" podUID="469a686d-3a72-4262-b5e2-1e5357ec90a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Aug 04 11:01:46.653860 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:46.653821 2580 patch_prober.go:28] interesting pod/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Aug 04 11:01:46.654067 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:46.653899 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" podUID="469a686d-3a72-4262-b5e2-1e5357ec90a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Aug 04 11:01:47.657257 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:47.657222 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:47.657759 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:47.657591 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:47.658302 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:47.658280 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:01:55.936683 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:01:55.936648 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:02:01.056528 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:01.056484 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw"] Aug 04 11:02:01.056992 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:01.056746 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" podUID="469a686d-3a72-4262-b5e2-1e5357ec90a5" containerName="istio-proxy" containerID="cri-o://3fe8c771cb58a97da4cb59fb48feab01148728be1a38876690d4f3c54148f068" gracePeriod=30 Aug 04 11:02:02.659976 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:02.659938 2580 patch_prober.go:28] interesting pod/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Aug 04 11:02:02.660356 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:02.659999 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" podUID="469a686d-3a72-4262-b5e2-1e5357ec90a5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 11:02:06.302485 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.302458 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:02:06.404007 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.403975 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/469a686d-3a72-4262-b5e2-1e5357ec90a5-credential-socket\") pod \"469a686d-3a72-4262-b5e2-1e5357ec90a5\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " Aug 04 11:02:06.404007 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.404010 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/469a686d-3a72-4262-b5e2-1e5357ec90a5-istio-token\") pod \"469a686d-3a72-4262-b5e2-1e5357ec90a5\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " Aug 04 11:02:06.404256 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.404035 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/469a686d-3a72-4262-b5e2-1e5357ec90a5-istio-podinfo\") pod \"469a686d-3a72-4262-b5e2-1e5357ec90a5\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " Aug 04 11:02:06.404256 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.404059 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/469a686d-3a72-4262-b5e2-1e5357ec90a5-istio-envoy\") pod \"469a686d-3a72-4262-b5e2-1e5357ec90a5\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " Aug 04 11:02:06.404256 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.404083 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/469a686d-3a72-4262-b5e2-1e5357ec90a5-workload-certs\") pod \"469a686d-3a72-4262-b5e2-1e5357ec90a5\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " Aug 04 11:02:06.404256 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.404132 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/469a686d-3a72-4262-b5e2-1e5357ec90a5-workload-socket\") pod \"469a686d-3a72-4262-b5e2-1e5357ec90a5\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " Aug 04 11:02:06.404256 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.404159 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/469a686d-3a72-4262-b5e2-1e5357ec90a5-istio-data\") pod \"469a686d-3a72-4262-b5e2-1e5357ec90a5\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " Aug 04 11:02:06.404256 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.404185 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72q4m\" (UniqueName: \"kubernetes.io/projected/469a686d-3a72-4262-b5e2-1e5357ec90a5-kube-api-access-72q4m\") pod \"469a686d-3a72-4262-b5e2-1e5357ec90a5\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " Aug 04 11:02:06.404256 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.404242 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/469a686d-3a72-4262-b5e2-1e5357ec90a5-istiod-ca-cert\") pod \"469a686d-3a72-4262-b5e2-1e5357ec90a5\" (UID: \"469a686d-3a72-4262-b5e2-1e5357ec90a5\") " Aug 04 11:02:06.404620 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.404315 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469a686d-3a72-4262-b5e2-1e5357ec90a5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "469a686d-3a72-4262-b5e2-1e5357ec90a5" (UID: "469a686d-3a72-4262-b5e2-1e5357ec90a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:02:06.404620 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.404449 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469a686d-3a72-4262-b5e2-1e5357ec90a5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "469a686d-3a72-4262-b5e2-1e5357ec90a5" (UID: "469a686d-3a72-4262-b5e2-1e5357ec90a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:02:06.404620 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.404523 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469a686d-3a72-4262-b5e2-1e5357ec90a5-istio-data" (OuterVolumeSpecName: "istio-data") pod "469a686d-3a72-4262-b5e2-1e5357ec90a5" (UID: "469a686d-3a72-4262-b5e2-1e5357ec90a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:02:06.404620 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.404533 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/469a686d-3a72-4262-b5e2-1e5357ec90a5-credential-socket\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:02:06.404797 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.404750 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469a686d-3a72-4262-b5e2-1e5357ec90a5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "469a686d-3a72-4262-b5e2-1e5357ec90a5" (UID: "469a686d-3a72-4262-b5e2-1e5357ec90a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:02:06.404851 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.404829 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/469a686d-3a72-4262-b5e2-1e5357ec90a5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "469a686d-3a72-4262-b5e2-1e5357ec90a5" (UID: "469a686d-3a72-4262-b5e2-1e5357ec90a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:02:06.406401 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.406377 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/469a686d-3a72-4262-b5e2-1e5357ec90a5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "469a686d-3a72-4262-b5e2-1e5357ec90a5" (UID: "469a686d-3a72-4262-b5e2-1e5357ec90a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 11:02:06.406718 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.406697 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/469a686d-3a72-4262-b5e2-1e5357ec90a5-kube-api-access-72q4m" (OuterVolumeSpecName: "kube-api-access-72q4m") pod "469a686d-3a72-4262-b5e2-1e5357ec90a5" (UID: "469a686d-3a72-4262-b5e2-1e5357ec90a5"). InnerVolumeSpecName "kube-api-access-72q4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:02:06.406854 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.406828 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/469a686d-3a72-4262-b5e2-1e5357ec90a5-istio-token" (OuterVolumeSpecName: "istio-token") pod "469a686d-3a72-4262-b5e2-1e5357ec90a5" (UID: "469a686d-3a72-4262-b5e2-1e5357ec90a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:02:06.406950 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.406827 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469a686d-3a72-4262-b5e2-1e5357ec90a5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "469a686d-3a72-4262-b5e2-1e5357ec90a5" (UID: "469a686d-3a72-4262-b5e2-1e5357ec90a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:02:06.455034 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.455002 2580 generic.go:356] "Generic (PLEG): container finished" podID="469a686d-3a72-4262-b5e2-1e5357ec90a5" containerID="3fe8c771cb58a97da4cb59fb48feab01148728be1a38876690d4f3c54148f068" exitCode=0 Aug 04 11:02:06.455209 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.455045 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" event={"ID":"469a686d-3a72-4262-b5e2-1e5357ec90a5","Type":"ContainerDied","Data":"3fe8c771cb58a97da4cb59fb48feab01148728be1a38876690d4f3c54148f068"} Aug 04 11:02:06.455209 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.455049 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" Aug 04 11:02:06.455209 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.455089 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw" event={"ID":"469a686d-3a72-4262-b5e2-1e5357ec90a5","Type":"ContainerDied","Data":"e6c6d955d86a60adf2f6f36a3e127d6430617f9ba9de3f4deb86a6ca3946b0b9"} Aug 04 11:02:06.455209 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.455118 2580 scope.go:117] "RemoveContainer" containerID="3fe8c771cb58a97da4cb59fb48feab01148728be1a38876690d4f3c54148f068" Aug 04 11:02:06.464230 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.464210 2580 scope.go:117] "RemoveContainer" containerID="e2562b867db89dd98d24e835fc6b663a1c3042c2c2367ad6ac53c81d7ee5cb46" Aug 04 11:02:06.472252 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.472233 2580 scope.go:117] "RemoveContainer" containerID="3fe8c771cb58a97da4cb59fb48feab01148728be1a38876690d4f3c54148f068" Aug 04 11:02:06.472512 ip-10-0-142-113 kubenswrapper[2580]: E0804 11:02:06.472491 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fe8c771cb58a97da4cb59fb48feab01148728be1a38876690d4f3c54148f068\": container with ID starting with 3fe8c771cb58a97da4cb59fb48feab01148728be1a38876690d4f3c54148f068 not found: ID does not exist" containerID="3fe8c771cb58a97da4cb59fb48feab01148728be1a38876690d4f3c54148f068" Aug 04 11:02:06.472603 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.472520 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fe8c771cb58a97da4cb59fb48feab01148728be1a38876690d4f3c54148f068"} err="failed to get container status \"3fe8c771cb58a97da4cb59fb48feab01148728be1a38876690d4f3c54148f068\": rpc error: code = NotFound desc = could not find container \"3fe8c771cb58a97da4cb59fb48feab01148728be1a38876690d4f3c54148f068\": container with ID starting with 3fe8c771cb58a97da4cb59fb48feab01148728be1a38876690d4f3c54148f068 not found: ID does not exist" Aug 04 11:02:06.472603 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.472539 2580 scope.go:117] "RemoveContainer" containerID="e2562b867db89dd98d24e835fc6b663a1c3042c2c2367ad6ac53c81d7ee5cb46" Aug 04 11:02:06.472774 ip-10-0-142-113 kubenswrapper[2580]: E0804 11:02:06.472759 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2562b867db89dd98d24e835fc6b663a1c3042c2c2367ad6ac53c81d7ee5cb46\": container with ID starting with e2562b867db89dd98d24e835fc6b663a1c3042c2c2367ad6ac53c81d7ee5cb46 not found: ID does not exist" containerID="e2562b867db89dd98d24e835fc6b663a1c3042c2c2367ad6ac53c81d7ee5cb46" Aug 04 11:02:06.472816 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.472779 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2562b867db89dd98d24e835fc6b663a1c3042c2c2367ad6ac53c81d7ee5cb46"} err="failed to get container status \"e2562b867db89dd98d24e835fc6b663a1c3042c2c2367ad6ac53c81d7ee5cb46\": rpc error: code = NotFound desc = could not find container \"e2562b867db89dd98d24e835fc6b663a1c3042c2c2367ad6ac53c81d7ee5cb46\": container with ID starting with e2562b867db89dd98d24e835fc6b663a1c3042c2c2367ad6ac53c81d7ee5cb46 not found: ID does not exist" Aug 04 11:02:06.476439 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.476418 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw"] Aug 04 11:02:06.480605 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.480583 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-fe15eb98-openshift-default-5ff785f94d-9crjw"] Aug 04 11:02:06.505680 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.505659 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/469a686d-3a72-4262-b5e2-1e5357ec90a5-istio-token\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:02:06.505680 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.505683 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/469a686d-3a72-4262-b5e2-1e5357ec90a5-istio-podinfo\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:02:06.505825 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.505695 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/469a686d-3a72-4262-b5e2-1e5357ec90a5-istio-envoy\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:02:06.505825 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.505709 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/469a686d-3a72-4262-b5e2-1e5357ec90a5-workload-certs\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:02:06.505825 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.505718 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/469a686d-3a72-4262-b5e2-1e5357ec90a5-workload-socket\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:02:06.505825 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.505725 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/469a686d-3a72-4262-b5e2-1e5357ec90a5-istio-data\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:02:06.505825 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.505734 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72q4m\" (UniqueName: \"kubernetes.io/projected/469a686d-3a72-4262-b5e2-1e5357ec90a5-kube-api-access-72q4m\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:02:06.505825 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:06.505744 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/469a686d-3a72-4262-b5e2-1e5357ec90a5-istiod-ca-cert\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:02:07.099046 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:07.099010 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="469a686d-3a72-4262-b5e2-1e5357ec90a5" path="/var/lib/kubelet/pods/469a686d-3a72-4262-b5e2-1e5357ec90a5/volumes" Aug 04 11:02:36.030269 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:36.030232 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:02:41.159464 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:41.159438 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tb4zk_c5922147-cb42-4f62-9880-8deaa51a4b0f/ovn-acl-logging/0.log" Aug 04 11:02:41.164890 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:41.164874 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tb4zk_c5922147-cb42-4f62-9880-8deaa51a4b0f/ovn-acl-logging/0.log" Aug 04 11:02:42.972986 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:42.972944 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9t69l"] Aug 04 11:02:42.973481 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:42.973218 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9t69l" podUID="690f2b91-2762-4b2e-ac6c-561d645c73a6" containerName="manager" containerID="cri-o://632c0b35701ca7fa1d3f5497b3efa6f374fa90dbfe49cc4384ea80e52c26b937" gracePeriod=10 Aug 04 11:02:43.230320 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:43.230247 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9t69l" Aug 04 11:02:43.318266 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:43.318234 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlpxt\" (UniqueName: \"kubernetes.io/projected/690f2b91-2762-4b2e-ac6c-561d645c73a6-kube-api-access-mlpxt\") pod \"690f2b91-2762-4b2e-ac6c-561d645c73a6\" (UID: \"690f2b91-2762-4b2e-ac6c-561d645c73a6\") " Aug 04 11:02:43.318447 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:43.318320 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/690f2b91-2762-4b2e-ac6c-561d645c73a6-extensions-socket-volume\") pod \"690f2b91-2762-4b2e-ac6c-561d645c73a6\" (UID: \"690f2b91-2762-4b2e-ac6c-561d645c73a6\") " Aug 04 11:02:43.318679 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:43.318658 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/690f2b91-2762-4b2e-ac6c-561d645c73a6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "690f2b91-2762-4b2e-ac6c-561d645c73a6" (UID: "690f2b91-2762-4b2e-ac6c-561d645c73a6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:02:43.320505 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:43.320485 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/690f2b91-2762-4b2e-ac6c-561d645c73a6-kube-api-access-mlpxt" (OuterVolumeSpecName: "kube-api-access-mlpxt") pod "690f2b91-2762-4b2e-ac6c-561d645c73a6" (UID: "690f2b91-2762-4b2e-ac6c-561d645c73a6"). InnerVolumeSpecName "kube-api-access-mlpxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:02:43.419064 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:43.419019 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlpxt\" (UniqueName: \"kubernetes.io/projected/690f2b91-2762-4b2e-ac6c-561d645c73a6-kube-api-access-mlpxt\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:02:43.419064 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:43.419050 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/690f2b91-2762-4b2e-ac6c-561d645c73a6-extensions-socket-volume\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:02:43.621286 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:43.621256 2580 generic.go:356] "Generic (PLEG): container finished" podID="690f2b91-2762-4b2e-ac6c-561d645c73a6" containerID="632c0b35701ca7fa1d3f5497b3efa6f374fa90dbfe49cc4384ea80e52c26b937" exitCode=0 Aug 04 11:02:43.621512 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:43.621305 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9t69l" Aug 04 11:02:43.621512 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:43.621343 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9t69l" event={"ID":"690f2b91-2762-4b2e-ac6c-561d645c73a6","Type":"ContainerDied","Data":"632c0b35701ca7fa1d3f5497b3efa6f374fa90dbfe49cc4384ea80e52c26b937"} Aug 04 11:02:43.621512 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:43.621402 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9t69l" event={"ID":"690f2b91-2762-4b2e-ac6c-561d645c73a6","Type":"ContainerDied","Data":"09458021610fc24a0c2aca47dda79bd017c9f2bab4be71cd2293d8cc49bb87bd"} Aug 04 11:02:43.621512 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:43.621438 2580 scope.go:117] "RemoveContainer" containerID="632c0b35701ca7fa1d3f5497b3efa6f374fa90dbfe49cc4384ea80e52c26b937" Aug 04 11:02:43.631782 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:43.631758 2580 scope.go:117] "RemoveContainer" containerID="632c0b35701ca7fa1d3f5497b3efa6f374fa90dbfe49cc4384ea80e52c26b937" Aug 04 11:02:43.632063 ip-10-0-142-113 kubenswrapper[2580]: E0804 11:02:43.632044 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"632c0b35701ca7fa1d3f5497b3efa6f374fa90dbfe49cc4384ea80e52c26b937\": container with ID starting with 632c0b35701ca7fa1d3f5497b3efa6f374fa90dbfe49cc4384ea80e52c26b937 not found: ID does not exist" containerID="632c0b35701ca7fa1d3f5497b3efa6f374fa90dbfe49cc4384ea80e52c26b937" Aug 04 11:02:43.632114 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:43.632073 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"632c0b35701ca7fa1d3f5497b3efa6f374fa90dbfe49cc4384ea80e52c26b937"} err="failed to get container status \"632c0b35701ca7fa1d3f5497b3efa6f374fa90dbfe49cc4384ea80e52c26b937\": rpc error: code = NotFound desc = could not find container \"632c0b35701ca7fa1d3f5497b3efa6f374fa90dbfe49cc4384ea80e52c26b937\": container with ID starting with 632c0b35701ca7fa1d3f5497b3efa6f374fa90dbfe49cc4384ea80e52c26b937 not found: ID does not exist" Aug 04 11:02:43.645476 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:43.645451 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9t69l"] Aug 04 11:02:43.648802 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:43.648781 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9t69l"] Aug 04 11:02:45.100452 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:02:45.100416 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="690f2b91-2762-4b2e-ac6c-561d645c73a6" path="/var/lib/kubelet/pods/690f2b91-2762-4b2e-ac6c-561d645c73a6/volumes" Aug 04 11:03:58.576512 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:03:58.576476 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-h56g4"] Aug 04 11:03:58.576941 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:03:58.576807 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4246b32-957a-4473-b784-5fc5174e24e2" containerName="istio-proxy" Aug 04 11:03:58.576941 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:03:58.576818 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4246b32-957a-4473-b784-5fc5174e24e2" containerName="istio-proxy" Aug 04 11:03:58.576941 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:03:58.576830 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="469a686d-3a72-4262-b5e2-1e5357ec90a5" containerName="istio-proxy" Aug 04 11:03:58.576941 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:03:58.576836 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="469a686d-3a72-4262-b5e2-1e5357ec90a5" containerName="istio-proxy" Aug 04 11:03:58.576941 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:03:58.576845 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="469a686d-3a72-4262-b5e2-1e5357ec90a5" containerName="istio-proxy" Aug 04 11:03:58.576941 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:03:58.576851 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="469a686d-3a72-4262-b5e2-1e5357ec90a5" containerName="istio-proxy" Aug 04 11:03:58.576941 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:03:58.576865 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="690f2b91-2762-4b2e-ac6c-561d645c73a6" containerName="manager" Aug 04 11:03:58.576941 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:03:58.576870 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="690f2b91-2762-4b2e-ac6c-561d645c73a6" containerName="manager" Aug 04 11:03:58.576941 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:03:58.576933 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f694f65a-c102-47ed-8a3d-943448d04892" containerName="istio-proxy" Aug 04 11:03:58.576941 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:03:58.576942 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="690f2b91-2762-4b2e-ac6c-561d645c73a6" containerName="manager" Aug 04 11:03:58.577239 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:03:58.576949 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="469a686d-3a72-4262-b5e2-1e5357ec90a5" containerName="istio-proxy" Aug 04 11:03:58.577239 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:03:58.576957 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="469a686d-3a72-4262-b5e2-1e5357ec90a5" containerName="istio-proxy" Aug 04 11:03:58.579570 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:03:58.579540 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-h56g4" Aug 04 11:03:58.582447 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:03:58.582431 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-spfp7\"" Aug 04 11:03:58.590036 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:03:58.590013 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-h56g4"] Aug 04 11:03:58.650517 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:03:58.650480 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/96ffd714-48bf-432c-8ce1-112fb49eea4a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-h56g4\" (UID: \"96ffd714-48bf-432c-8ce1-112fb49eea4a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-h56g4" Aug 04 11:03:58.650702 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:03:58.650527 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd6lx\" (UniqueName: \"kubernetes.io/projected/96ffd714-48bf-432c-8ce1-112fb49eea4a-kube-api-access-kd6lx\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-h56g4\" (UID: \"96ffd714-48bf-432c-8ce1-112fb49eea4a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-h56g4" Aug 04 11:03:58.751440 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:03:58.751401 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd6lx\" (UniqueName: \"kubernetes.io/projected/96ffd714-48bf-432c-8ce1-112fb49eea4a-kube-api-access-kd6lx\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-h56g4\" (UID: \"96ffd714-48bf-432c-8ce1-112fb49eea4a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-h56g4" Aug 04 11:03:58.751670 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:03:58.751479 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/96ffd714-48bf-432c-8ce1-112fb49eea4a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-h56g4\" (UID: \"96ffd714-48bf-432c-8ce1-112fb49eea4a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-h56g4" Aug 04 11:03:58.751846 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:03:58.751827 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/96ffd714-48bf-432c-8ce1-112fb49eea4a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-h56g4\" (UID: \"96ffd714-48bf-432c-8ce1-112fb49eea4a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-h56g4" Aug 04 11:03:58.760971 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:03:58.760941 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd6lx\" (UniqueName: \"kubernetes.io/projected/96ffd714-48bf-432c-8ce1-112fb49eea4a-kube-api-access-kd6lx\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-h56g4\" (UID: \"96ffd714-48bf-432c-8ce1-112fb49eea4a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-h56g4" Aug 04 11:03:58.889722 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:03:58.889681 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-h56g4" Aug 04 11:03:59.033811 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:03:59.033781 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-h56g4"] Aug 04 11:03:59.036018 ip-10-0-142-113 kubenswrapper[2580]: W0804 11:03:59.035986 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96ffd714_48bf_432c_8ce1_112fb49eea4a.slice/crio-d6a6db15fb815b984ef8f352fa0538ac2b37ee9eb6712c551f82e5d275253e6e WatchSource:0}: Error finding container d6a6db15fb815b984ef8f352fa0538ac2b37ee9eb6712c551f82e5d275253e6e: Status 404 returned error can't find the container with id d6a6db15fb815b984ef8f352fa0538ac2b37ee9eb6712c551f82e5d275253e6e Aug 04 11:03:59.959840 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:03:59.959797 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-h56g4" event={"ID":"96ffd714-48bf-432c-8ce1-112fb49eea4a","Type":"ContainerStarted","Data":"1386483de04e4e7dd6fae2d6f1fcc6a64351722830c74832b213be92a61d37b3"} Aug 04 11:03:59.960212 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:03:59.959898 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-h56g4" event={"ID":"96ffd714-48bf-432c-8ce1-112fb49eea4a","Type":"ContainerStarted","Data":"d6a6db15fb815b984ef8f352fa0538ac2b37ee9eb6712c551f82e5d275253e6e"} Aug 04 11:03:59.960212 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:03:59.959944 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-h56g4" Aug 04 11:03:59.990543 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:03:59.990488 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-h56g4" podStartSLOduration=1.990472577 podStartE2EDuration="1.990472577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:03:59.989130904 +0000 UTC m=+2179.410018316" watchObservedRunningTime="2026-08-04 11:03:59.990472577 +0000 UTC m=+2179.411359988" Aug 04 11:04:10.967382 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:10.967350 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-h56g4" Aug 04 11:04:17.884927 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:17.884892 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd"] Aug 04 11:04:17.888080 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:17.888061 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:17.890866 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:17.890846 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a18af8fb-openshift-default-dockercfg-bgrrj\"" Aug 04 11:04:17.900814 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:17.900784 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd"] Aug 04 11:04:18.019287 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:18.019247 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3c6157d-9d6d-4fd1-931d-d9487d43a461-istio-token\") pod \"e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:18.019488 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:18.019293 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3c6157d-9d6d-4fd1-931d-d9487d43a461-istiod-ca-cert\") pod \"e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:18.019488 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:18.019371 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c6157d-9d6d-4fd1-931d-d9487d43a461-credential-socket\") pod \"e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:18.019488 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:18.019392 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcvlz\" (UniqueName: \"kubernetes.io/projected/b3c6157d-9d6d-4fd1-931d-d9487d43a461-kube-api-access-jcvlz\") pod \"e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:18.019488 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:18.019423 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3c6157d-9d6d-4fd1-931d-d9487d43a461-workload-certs\") pod \"e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:18.019488 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:18.019455 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3c6157d-9d6d-4fd1-931d-d9487d43a461-istio-data\") pod \"e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:18.019704 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:18.019533 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3c6157d-9d6d-4fd1-931d-d9487d43a461-istio-podinfo\") pod \"e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:18.019704 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:18.019595 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3c6157d-9d6d-4fd1-931d-d9487d43a461-istio-envoy\") pod \"e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:18.019704 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:18.019636 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c6157d-9d6d-4fd1-931d-d9487d43a461-workload-socket\") pod \"e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:18.120252 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:18.120213 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3c6157d-9d6d-4fd1-931d-d9487d43a461-istio-envoy\") pod \"e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:18.120252 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:18.120254 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c6157d-9d6d-4fd1-931d-d9487d43a461-workload-socket\") pod \"e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:18.120507 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:18.120281 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3c6157d-9d6d-4fd1-931d-d9487d43a461-istio-token\") pod \"e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:18.120507 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:18.120301 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3c6157d-9d6d-4fd1-931d-d9487d43a461-istiod-ca-cert\") pod \"e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:18.120507 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:18.120336 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c6157d-9d6d-4fd1-931d-d9487d43a461-credential-socket\") pod \"e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:18.120507 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:18.120354 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcvlz\" (UniqueName: \"kubernetes.io/projected/b3c6157d-9d6d-4fd1-931d-d9487d43a461-kube-api-access-jcvlz\") pod \"e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:18.120507 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:18.120373 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3c6157d-9d6d-4fd1-931d-d9487d43a461-workload-certs\") pod \"e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:18.120507 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:18.120399 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3c6157d-9d6d-4fd1-931d-d9487d43a461-istio-data\") pod \"e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:18.120507 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:18.120476 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3c6157d-9d6d-4fd1-931d-d9487d43a461-istio-podinfo\") pod \"e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:18.120908 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:18.120807 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c6157d-9d6d-4fd1-931d-d9487d43a461-workload-socket\") pod \"e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:18.120966 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:18.120905 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3c6157d-9d6d-4fd1-931d-d9487d43a461-workload-certs\") pod \"e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:18.121024 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:18.120979 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c6157d-9d6d-4fd1-931d-d9487d43a461-credential-socket\") pod \"e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:18.121024 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:18.120995 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3c6157d-9d6d-4fd1-931d-d9487d43a461-istio-data\") pod \"e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:18.121153 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:18.121135 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3c6157d-9d6d-4fd1-931d-d9487d43a461-istiod-ca-cert\") pod \"e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:18.122740 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:18.122718 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3c6157d-9d6d-4fd1-931d-d9487d43a461-istio-envoy\") pod \"e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:18.122982 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:18.122964 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3c6157d-9d6d-4fd1-931d-d9487d43a461-istio-podinfo\") pod \"e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:18.128888 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:18.128857 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3c6157d-9d6d-4fd1-931d-d9487d43a461-istio-token\") pod \"e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:18.128981 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:18.128946 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcvlz\" (UniqueName: \"kubernetes.io/projected/b3c6157d-9d6d-4fd1-931d-d9487d43a461-kube-api-access-jcvlz\") pod \"e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:18.198221 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:18.198131 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:18.346331 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:18.346306 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd"] Aug 04 11:04:18.348428 ip-10-0-142-113 kubenswrapper[2580]: W0804 11:04:18.348398 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3c6157d_9d6d_4fd1_931d_d9487d43a461.slice/crio-7ba7ca7547f6de2d46f2a55657e7c695df03f84a84f41ccc2f44438e47bd2e55 WatchSource:0}: Error finding container 7ba7ca7547f6de2d46f2a55657e7c695df03f84a84f41ccc2f44438e47bd2e55: Status 404 returned error can't find the container with id 7ba7ca7547f6de2d46f2a55657e7c695df03f84a84f41ccc2f44438e47bd2e55 Aug 04 11:04:18.350837 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:18.350805 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:04:18.350945 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:18.350886 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:04:18.350945 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:18.350932 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:04:19.048354 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:19.048312 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" event={"ID":"b3c6157d-9d6d-4fd1-931d-d9487d43a461","Type":"ContainerStarted","Data":"5619a393666e208529b318d6401d8f6ef8eb03cd7e405e9edc9b5fccc466f5f1"} Aug 04 11:04:19.048780 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:19.048369 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" event={"ID":"b3c6157d-9d6d-4fd1-931d-d9487d43a461","Type":"ContainerStarted","Data":"7ba7ca7547f6de2d46f2a55657e7c695df03f84a84f41ccc2f44438e47bd2e55"} Aug 04 11:04:19.074673 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:19.074615 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" podStartSLOduration=2.074594252 podStartE2EDuration="2.074594252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:04:19.073225972 +0000 UTC m=+2198.494113384" watchObservedRunningTime="2026-08-04 11:04:19.074594252 +0000 UTC m=+2198.495481647" Aug 04 11:04:19.199291 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:19.199255 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:19.204512 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:19.204482 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:20.053734 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:20.053702 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:20.054669 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:20.054648 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:21.328408 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:21.328359 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:04:28.840525 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:28.840486 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:04:35.054738 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:35.054711 2580 patch_prober.go:28] interesting pod/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Aug 04 11:04:35.055080 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:35.054779 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" podUID="b3c6157d-9d6d-4fd1-931d-d9487d43a461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Aug 04 11:04:35.137274 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:35.137242 2580 generic.go:356] "Generic (PLEG): container finished" podID="b3c6157d-9d6d-4fd1-931d-d9487d43a461" containerID="5619a393666e208529b318d6401d8f6ef8eb03cd7e405e9edc9b5fccc466f5f1" exitCode=0 Aug 04 11:04:35.137455 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:35.137315 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" event={"ID":"b3c6157d-9d6d-4fd1-931d-d9487d43a461","Type":"ContainerDied","Data":"5619a393666e208529b318d6401d8f6ef8eb03cd7e405e9edc9b5fccc466f5f1"} Aug 04 11:04:35.137743 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:35.137730 2580 scope.go:117] "RemoveContainer" containerID="5619a393666e208529b318d6401d8f6ef8eb03cd7e405e9edc9b5fccc466f5f1" Aug 04 11:04:35.138827 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:35.138797 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:04:35.138905 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:35.138887 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:04:35.138947 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:35.138936 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:04:36.143910 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:36.143876 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" event={"ID":"b3c6157d-9d6d-4fd1-931d-d9487d43a461","Type":"ContainerStarted","Data":"1d888c7c1b675a42d0ebd92a6a9b3c76ef3bc60f366e8d340ccd1592cd73464f"} Aug 04 11:04:36.199097 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:36.199056 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:36.199398 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:36.199105 2580 patch_prober.go:28] interesting pod/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Aug 04 11:04:36.199617 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:36.199553 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" podUID="b3c6157d-9d6d-4fd1-931d-d9487d43a461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Aug 04 11:04:37.199500 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:37.199461 2580 patch_prober.go:28] interesting pod/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Aug 04 11:04:37.200018 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:37.199654 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" podUID="b3c6157d-9d6d-4fd1-931d-d9487d43a461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Aug 04 11:04:38.199402 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:38.199350 2580 patch_prober.go:28] interesting pod/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Aug 04 11:04:38.199638 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:38.199439 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" podUID="b3c6157d-9d6d-4fd1-931d-d9487d43a461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Aug 04 11:04:39.198891 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:39.198852 2580 patch_prober.go:28] interesting pod/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Aug 04 11:04:39.199192 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:39.199162 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" podUID="b3c6157d-9d6d-4fd1-931d-d9487d43a461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Aug 04 11:04:40.199393 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:40.198935 2580 patch_prober.go:28] interesting pod/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Aug 04 11:04:40.199393 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:40.199004 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" podUID="b3c6157d-9d6d-4fd1-931d-d9487d43a461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Aug 04 11:04:41.203454 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:41.203420 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:41.203829 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:41.203814 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:04:41.204537 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:04:41.204517 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:05:03.247019 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:03.246938 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h"] Aug 04 11:05:03.249475 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:03.249453 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" Aug 04 11:05:03.253933 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:03.253904 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-a18af8fb\"/\"openshift-service-ca.crt\"" Aug 04 11:05:03.254072 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:03.253950 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-a18af8fb\"/\"e2e-lifecycle-model-a18af8fb-kserve-self-signed-certs\"" Aug 04 11:05:03.254072 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:03.253961 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-a18af8fb\"/\"kube-root-ca.crt\"" Aug 04 11:05:03.254072 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:03.253951 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-a18af8fb\"/\"default-dockercfg-fj9tr\"" Aug 04 11:05:03.259185 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:03.259162 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h"] Aug 04 11:05:03.324588 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:03.324533 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af15cace-fad3-44fe-8824-23d57674220d-tls-certs\") pod \"e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h\" (UID: \"af15cace-fad3-44fe-8824-23d57674220d\") " pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" Aug 04 11:05:03.324773 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:03.324595 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqfpq\" (UniqueName: \"kubernetes.io/projected/af15cace-fad3-44fe-8824-23d57674220d-kube-api-access-nqfpq\") pod \"e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h\" (UID: \"af15cace-fad3-44fe-8824-23d57674220d\") " pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" Aug 04 11:05:03.324773 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:03.324674 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af15cace-fad3-44fe-8824-23d57674220d-home\") pod \"e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h\" (UID: \"af15cace-fad3-44fe-8824-23d57674220d\") " pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" Aug 04 11:05:03.324773 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:03.324703 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af15cace-fad3-44fe-8824-23d57674220d-dshm\") pod \"e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h\" (UID: \"af15cace-fad3-44fe-8824-23d57674220d\") " pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" Aug 04 11:05:03.324773 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:03.324733 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af15cace-fad3-44fe-8824-23d57674220d-tmp-dir\") pod \"e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h\" (UID: \"af15cace-fad3-44fe-8824-23d57674220d\") " pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" Aug 04 11:05:03.324941 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:03.324846 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af15cace-fad3-44fe-8824-23d57674220d-model-cache\") pod \"e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h\" (UID: \"af15cace-fad3-44fe-8824-23d57674220d\") " pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" Aug 04 11:05:03.425648 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:03.425609 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af15cace-fad3-44fe-8824-23d57674220d-model-cache\") pod \"e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h\" (UID: \"af15cace-fad3-44fe-8824-23d57674220d\") " pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" Aug 04 11:05:03.425850 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:03.425662 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af15cace-fad3-44fe-8824-23d57674220d-tls-certs\") pod \"e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h\" (UID: \"af15cace-fad3-44fe-8824-23d57674220d\") " pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" Aug 04 11:05:03.425850 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:03.425693 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqfpq\" (UniqueName: \"kubernetes.io/projected/af15cace-fad3-44fe-8824-23d57674220d-kube-api-access-nqfpq\") pod \"e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h\" (UID: \"af15cace-fad3-44fe-8824-23d57674220d\") " pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" Aug 04 11:05:03.425850 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:03.425752 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af15cace-fad3-44fe-8824-23d57674220d-home\") pod \"e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h\" (UID: \"af15cace-fad3-44fe-8824-23d57674220d\") " pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" Aug 04 11:05:03.426034 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:03.425876 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af15cace-fad3-44fe-8824-23d57674220d-dshm\") pod \"e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h\" (UID: \"af15cace-fad3-44fe-8824-23d57674220d\") " pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" Aug 04 11:05:03.426034 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:03.425920 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af15cace-fad3-44fe-8824-23d57674220d-tmp-dir\") pod \"e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h\" (UID: \"af15cace-fad3-44fe-8824-23d57674220d\") " pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" Aug 04 11:05:03.426138 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:03.426055 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af15cace-fad3-44fe-8824-23d57674220d-model-cache\") pod \"e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h\" (UID: \"af15cace-fad3-44fe-8824-23d57674220d\") " pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" Aug 04 11:05:03.426138 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:03.426089 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af15cace-fad3-44fe-8824-23d57674220d-home\") pod \"e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h\" (UID: \"af15cace-fad3-44fe-8824-23d57674220d\") " pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" Aug 04 11:05:03.426251 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:03.426234 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af15cace-fad3-44fe-8824-23d57674220d-tmp-dir\") pod \"e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h\" (UID: \"af15cace-fad3-44fe-8824-23d57674220d\") " pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" Aug 04 11:05:03.428245 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:03.428224 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af15cace-fad3-44fe-8824-23d57674220d-dshm\") pod \"e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h\" (UID: \"af15cace-fad3-44fe-8824-23d57674220d\") " pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" Aug 04 11:05:03.428423 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:03.428406 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af15cace-fad3-44fe-8824-23d57674220d-tls-certs\") pod \"e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h\" (UID: \"af15cace-fad3-44fe-8824-23d57674220d\") " pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" Aug 04 11:05:03.435104 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:03.435079 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqfpq\" (UniqueName: \"kubernetes.io/projected/af15cace-fad3-44fe-8824-23d57674220d-kube-api-access-nqfpq\") pod \"e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h\" (UID: \"af15cace-fad3-44fe-8824-23d57674220d\") " pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" Aug 04 11:05:03.559011 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:03.558902 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" Aug 04 11:05:03.703846 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:03.703726 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h"] Aug 04 11:05:03.706221 ip-10-0-142-113 kubenswrapper[2580]: W0804 11:05:03.706189 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf15cace_fad3_44fe_8824_23d57674220d.slice/crio-bd031ac81964e57e6b20ab06345c4c06b1d1b86f3e36042c152aa35397f17690 WatchSource:0}: Error finding container bd031ac81964e57e6b20ab06345c4c06b1d1b86f3e36042c152aa35397f17690: Status 404 returned error can't find the container with id bd031ac81964e57e6b20ab06345c4c06b1d1b86f3e36042c152aa35397f17690 Aug 04 11:05:04.272016 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:04.271975 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" event={"ID":"af15cace-fad3-44fe-8824-23d57674220d","Type":"ContainerStarted","Data":"865a99f691c9f31ab74d8405d7659a4dcf0bb06ccbc5358c6760c51ce351cdc9"} Aug 04 11:05:04.272412 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:04.272028 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" event={"ID":"af15cace-fad3-44fe-8824-23d57674220d","Type":"ContainerStarted","Data":"bd031ac81964e57e6b20ab06345c4c06b1d1b86f3e36042c152aa35397f17690"} Aug 04 11:05:05.535486 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:05.535428 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" podStartSLOduration=2.535408837 podStartE2EDuration="2.535408837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:05:04.290937617 +0000 UTC m=+2243.711825030" watchObservedRunningTime="2026-08-04 11:05:05.535408837 +0000 UTC m=+2244.956296275" Aug 04 11:05:05.536584 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:05.536548 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:05:13.560029 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:13.559989 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" Aug 04 11:05:13.560416 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:13.560043 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" Aug 04 11:05:13.572527 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:13.572500 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" Aug 04 11:05:14.330321 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:14.330278 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" Aug 04 11:05:20.140292 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:20.140253 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:05:38.442105 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:38.442069 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:05:46.230303 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:46.230267 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:05:49.176735 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:05:49.176696 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h"] Aug 04 11:06:04.196614 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:04.196529 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" podUID="af15cace-fad3-44fe-8824-23d57674220d" containerName="main" containerID="cri-o://865a99f691c9f31ab74d8405d7659a4dcf0bb06ccbc5358c6760c51ce351cdc9" gracePeriod=45 Aug 04 11:06:04.319725 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:04.319683 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" podUID="af15cace-fad3-44fe-8824-23d57674220d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": dial tcp 10.133.0.46:8000: connect: connection refused" Aug 04 11:06:04.458847 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:04.458782 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" Aug 04 11:06:04.553845 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:04.553810 2580 generic.go:356] "Generic (PLEG): container finished" podID="af15cace-fad3-44fe-8824-23d57674220d" containerID="865a99f691c9f31ab74d8405d7659a4dcf0bb06ccbc5358c6760c51ce351cdc9" exitCode=0 Aug 04 11:06:04.553845 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:04.553850 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" Aug 04 11:06:04.554064 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:04.553894 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" event={"ID":"af15cace-fad3-44fe-8824-23d57674220d","Type":"ContainerDied","Data":"865a99f691c9f31ab74d8405d7659a4dcf0bb06ccbc5358c6760c51ce351cdc9"} Aug 04 11:06:04.554064 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:04.553933 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h" event={"ID":"af15cace-fad3-44fe-8824-23d57674220d","Type":"ContainerDied","Data":"bd031ac81964e57e6b20ab06345c4c06b1d1b86f3e36042c152aa35397f17690"} Aug 04 11:06:04.554064 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:04.553955 2580 scope.go:117] "RemoveContainer" containerID="865a99f691c9f31ab74d8405d7659a4dcf0bb06ccbc5358c6760c51ce351cdc9" Aug 04 11:06:04.564104 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:04.564084 2580 scope.go:117] "RemoveContainer" containerID="865a99f691c9f31ab74d8405d7659a4dcf0bb06ccbc5358c6760c51ce351cdc9" Aug 04 11:06:04.564446 ip-10-0-142-113 kubenswrapper[2580]: E0804 11:06:04.564426 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"865a99f691c9f31ab74d8405d7659a4dcf0bb06ccbc5358c6760c51ce351cdc9\": container with ID starting with 865a99f691c9f31ab74d8405d7659a4dcf0bb06ccbc5358c6760c51ce351cdc9 not found: ID does not exist" containerID="865a99f691c9f31ab74d8405d7659a4dcf0bb06ccbc5358c6760c51ce351cdc9" Aug 04 11:06:04.564514 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:04.564461 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"865a99f691c9f31ab74d8405d7659a4dcf0bb06ccbc5358c6760c51ce351cdc9"} err="failed to get container status \"865a99f691c9f31ab74d8405d7659a4dcf0bb06ccbc5358c6760c51ce351cdc9\": rpc error: code = NotFound desc = could not find container \"865a99f691c9f31ab74d8405d7659a4dcf0bb06ccbc5358c6760c51ce351cdc9\": container with ID starting with 865a99f691c9f31ab74d8405d7659a4dcf0bb06ccbc5358c6760c51ce351cdc9 not found: ID does not exist" Aug 04 11:06:04.579940 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:04.579916 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af15cace-fad3-44fe-8824-23d57674220d-dshm\") pod \"af15cace-fad3-44fe-8824-23d57674220d\" (UID: \"af15cace-fad3-44fe-8824-23d57674220d\") " Aug 04 11:06:04.580058 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:04.579970 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af15cace-fad3-44fe-8824-23d57674220d-home\") pod \"af15cace-fad3-44fe-8824-23d57674220d\" (UID: \"af15cace-fad3-44fe-8824-23d57674220d\") " Aug 04 11:06:04.580058 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:04.580013 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af15cace-fad3-44fe-8824-23d57674220d-tls-certs\") pod \"af15cace-fad3-44fe-8824-23d57674220d\" (UID: \"af15cace-fad3-44fe-8824-23d57674220d\") " Aug 04 11:06:04.580058 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:04.580052 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af15cace-fad3-44fe-8824-23d57674220d-model-cache\") pod \"af15cace-fad3-44fe-8824-23d57674220d\" (UID: \"af15cace-fad3-44fe-8824-23d57674220d\") " Aug 04 11:06:04.580212 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:04.580089 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af15cace-fad3-44fe-8824-23d57674220d-tmp-dir\") pod \"af15cace-fad3-44fe-8824-23d57674220d\" (UID: \"af15cace-fad3-44fe-8824-23d57674220d\") " Aug 04 11:06:04.580212 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:04.580176 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqfpq\" (UniqueName: \"kubernetes.io/projected/af15cace-fad3-44fe-8824-23d57674220d-kube-api-access-nqfpq\") pod \"af15cace-fad3-44fe-8824-23d57674220d\" (UID: \"af15cace-fad3-44fe-8824-23d57674220d\") " Aug 04 11:06:04.580313 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:04.580268 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af15cace-fad3-44fe-8824-23d57674220d-home" (OuterVolumeSpecName: "home") pod "af15cace-fad3-44fe-8824-23d57674220d" (UID: "af15cace-fad3-44fe-8824-23d57674220d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:06:04.580375 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:04.580347 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af15cace-fad3-44fe-8824-23d57674220d-model-cache" (OuterVolumeSpecName: "model-cache") pod "af15cace-fad3-44fe-8824-23d57674220d" (UID: "af15cace-fad3-44fe-8824-23d57674220d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:06:04.580445 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:04.580422 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af15cace-fad3-44fe-8824-23d57674220d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "af15cace-fad3-44fe-8824-23d57674220d" (UID: "af15cace-fad3-44fe-8824-23d57674220d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:06:04.580537 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:04.580521 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af15cace-fad3-44fe-8824-23d57674220d-home\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:06:04.580629 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:04.580541 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af15cace-fad3-44fe-8824-23d57674220d-model-cache\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:06:04.580629 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:04.580602 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af15cace-fad3-44fe-8824-23d57674220d-tmp-dir\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:06:04.582300 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:04.582275 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af15cace-fad3-44fe-8824-23d57674220d-dshm" (OuterVolumeSpecName: "dshm") pod "af15cace-fad3-44fe-8824-23d57674220d" (UID: "af15cace-fad3-44fe-8824-23d57674220d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:06:04.582401 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:04.582361 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af15cace-fad3-44fe-8824-23d57674220d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "af15cace-fad3-44fe-8824-23d57674220d" (UID: "af15cace-fad3-44fe-8824-23d57674220d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:06:04.582401 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:04.582366 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af15cace-fad3-44fe-8824-23d57674220d-kube-api-access-nqfpq" (OuterVolumeSpecName: "kube-api-access-nqfpq") pod "af15cace-fad3-44fe-8824-23d57674220d" (UID: "af15cace-fad3-44fe-8824-23d57674220d"). InnerVolumeSpecName "kube-api-access-nqfpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:06:04.681311 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:04.681275 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqfpq\" (UniqueName: \"kubernetes.io/projected/af15cace-fad3-44fe-8824-23d57674220d-kube-api-access-nqfpq\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:06:04.681311 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:04.681306 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af15cace-fad3-44fe-8824-23d57674220d-dshm\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:06:04.681311 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:04.681317 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af15cace-fad3-44fe-8824-23d57674220d-tls-certs\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:06:04.877166 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:04.877133 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h"] Aug 04 11:06:04.881073 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:04.881043 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-a18af8fb/e2e-lifecycle-model-a18af8fb-kserve-576db84c9f-xfh9h"] Aug 04 11:06:05.104589 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:05.100832 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af15cace-fad3-44fe-8824-23d57674220d" path="/var/lib/kubelet/pods/af15cace-fad3-44fe-8824-23d57674220d/volumes" Aug 04 11:06:32.604216 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:32.604179 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd"] Aug 04 11:06:32.604765 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:32.604469 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" podUID="b3c6157d-9d6d-4fd1-931d-d9487d43a461" containerName="istio-proxy" containerID="cri-o://1d888c7c1b675a42d0ebd92a6a9b3c76ef3bc60f366e8d340ccd1592cd73464f" gracePeriod=30 Aug 04 11:06:34.637503 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:34.637465 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:06:37.861912 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:37.861885 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:06:37.957850 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:37.957761 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3c6157d-9d6d-4fd1-931d-d9487d43a461-istio-data\") pod \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " Aug 04 11:06:37.957850 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:37.957815 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3c6157d-9d6d-4fd1-931d-d9487d43a461-istiod-ca-cert\") pod \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " Aug 04 11:06:37.958082 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:37.957861 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c6157d-9d6d-4fd1-931d-d9487d43a461-credential-socket\") pod \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " Aug 04 11:06:37.958082 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:37.957883 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3c6157d-9d6d-4fd1-931d-d9487d43a461-istio-envoy\") pod \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " Aug 04 11:06:37.958082 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:37.957921 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcvlz\" (UniqueName: \"kubernetes.io/projected/b3c6157d-9d6d-4fd1-931d-d9487d43a461-kube-api-access-jcvlz\") pod \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " Aug 04 11:06:37.958082 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:37.957949 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3c6157d-9d6d-4fd1-931d-d9487d43a461-workload-certs\") pod \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " Aug 04 11:06:37.958082 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:37.957981 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c6157d-9d6d-4fd1-931d-d9487d43a461-workload-socket\") pod \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " Aug 04 11:06:37.958082 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:37.958017 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3c6157d-9d6d-4fd1-931d-d9487d43a461-istio-token\") pod \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " Aug 04 11:06:37.958082 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:37.958056 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3c6157d-9d6d-4fd1-931d-d9487d43a461-istio-podinfo\") pod \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\" (UID: \"b3c6157d-9d6d-4fd1-931d-d9487d43a461\") " Aug 04 11:06:37.958434 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:37.958100 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3c6157d-9d6d-4fd1-931d-d9487d43a461-istio-data" (OuterVolumeSpecName: "istio-data") pod "b3c6157d-9d6d-4fd1-931d-d9487d43a461" (UID: "b3c6157d-9d6d-4fd1-931d-d9487d43a461"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:06:37.958434 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:37.958196 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3c6157d-9d6d-4fd1-931d-d9487d43a461-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b3c6157d-9d6d-4fd1-931d-d9487d43a461" (UID: "b3c6157d-9d6d-4fd1-931d-d9487d43a461"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:06:37.958434 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:37.958281 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3c6157d-9d6d-4fd1-931d-d9487d43a461-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b3c6157d-9d6d-4fd1-931d-d9487d43a461" (UID: "b3c6157d-9d6d-4fd1-931d-d9487d43a461"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:06:37.958434 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:37.958306 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3c6157d-9d6d-4fd1-931d-d9487d43a461-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b3c6157d-9d6d-4fd1-931d-d9487d43a461" (UID: "b3c6157d-9d6d-4fd1-931d-d9487d43a461"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:06:37.958434 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:37.958306 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3c6157d-9d6d-4fd1-931d-d9487d43a461-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b3c6157d-9d6d-4fd1-931d-d9487d43a461" (UID: "b3c6157d-9d6d-4fd1-931d-d9487d43a461"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:06:37.958434 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:37.958372 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3c6157d-9d6d-4fd1-931d-d9487d43a461-istio-data\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:06:37.958434 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:37.958389 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c6157d-9d6d-4fd1-931d-d9487d43a461-credential-socket\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:06:37.958434 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:37.958399 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3c6157d-9d6d-4fd1-931d-d9487d43a461-workload-certs\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:06:37.960379 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:37.960344 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3c6157d-9d6d-4fd1-931d-d9487d43a461-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b3c6157d-9d6d-4fd1-931d-d9487d43a461" (UID: "b3c6157d-9d6d-4fd1-931d-d9487d43a461"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:06:37.960507 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:37.960412 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3c6157d-9d6d-4fd1-931d-d9487d43a461-kube-api-access-jcvlz" (OuterVolumeSpecName: "kube-api-access-jcvlz") pod "b3c6157d-9d6d-4fd1-931d-d9487d43a461" (UID: "b3c6157d-9d6d-4fd1-931d-d9487d43a461"). InnerVolumeSpecName "kube-api-access-jcvlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:06:37.960507 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:37.960451 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3c6157d-9d6d-4fd1-931d-d9487d43a461-istio-token" (OuterVolumeSpecName: "istio-token") pod "b3c6157d-9d6d-4fd1-931d-d9487d43a461" (UID: "b3c6157d-9d6d-4fd1-931d-d9487d43a461"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:06:37.960507 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:37.960456 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3c6157d-9d6d-4fd1-931d-d9487d43a461-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b3c6157d-9d6d-4fd1-931d-d9487d43a461" (UID: "b3c6157d-9d6d-4fd1-931d-d9487d43a461"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 11:06:38.059125 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:38.059086 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c6157d-9d6d-4fd1-931d-d9487d43a461-workload-socket\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:06:38.059125 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:38.059125 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3c6157d-9d6d-4fd1-931d-d9487d43a461-istio-token\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:06:38.059125 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:38.059135 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3c6157d-9d6d-4fd1-931d-d9487d43a461-istio-podinfo\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:06:38.059455 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:38.059148 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3c6157d-9d6d-4fd1-931d-d9487d43a461-istiod-ca-cert\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:06:38.059455 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:38.059161 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3c6157d-9d6d-4fd1-931d-d9487d43a461-istio-envoy\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:06:38.059455 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:38.059172 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcvlz\" (UniqueName: \"kubernetes.io/projected/b3c6157d-9d6d-4fd1-931d-d9487d43a461-kube-api-access-jcvlz\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:06:38.718396 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:38.718362 2580 generic.go:356] "Generic (PLEG): container finished" podID="b3c6157d-9d6d-4fd1-931d-d9487d43a461" containerID="1d888c7c1b675a42d0ebd92a6a9b3c76ef3bc60f366e8d340ccd1592cd73464f" exitCode=0 Aug 04 11:06:38.718600 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:38.718411 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" Aug 04 11:06:38.718600 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:38.718449 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" event={"ID":"b3c6157d-9d6d-4fd1-931d-d9487d43a461","Type":"ContainerDied","Data":"1d888c7c1b675a42d0ebd92a6a9b3c76ef3bc60f366e8d340ccd1592cd73464f"} Aug 04 11:06:38.718600 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:38.718510 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd" event={"ID":"b3c6157d-9d6d-4fd1-931d-d9487d43a461","Type":"ContainerDied","Data":"7ba7ca7547f6de2d46f2a55657e7c695df03f84a84f41ccc2f44438e47bd2e55"} Aug 04 11:06:38.718600 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:38.718535 2580 scope.go:117] "RemoveContainer" containerID="1d888c7c1b675a42d0ebd92a6a9b3c76ef3bc60f366e8d340ccd1592cd73464f" Aug 04 11:06:38.727967 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:38.727854 2580 scope.go:117] "RemoveContainer" containerID="5619a393666e208529b318d6401d8f6ef8eb03cd7e405e9edc9b5fccc466f5f1" Aug 04 11:06:38.737367 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:38.737263 2580 scope.go:117] "RemoveContainer" containerID="1d888c7c1b675a42d0ebd92a6a9b3c76ef3bc60f366e8d340ccd1592cd73464f" Aug 04 11:06:38.737915 ip-10-0-142-113 kubenswrapper[2580]: E0804 11:06:38.737890 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d888c7c1b675a42d0ebd92a6a9b3c76ef3bc60f366e8d340ccd1592cd73464f\": container with ID starting with 1d888c7c1b675a42d0ebd92a6a9b3c76ef3bc60f366e8d340ccd1592cd73464f not found: ID does not exist" containerID="1d888c7c1b675a42d0ebd92a6a9b3c76ef3bc60f366e8d340ccd1592cd73464f" Aug 04 11:06:38.738054 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:38.737926 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d888c7c1b675a42d0ebd92a6a9b3c76ef3bc60f366e8d340ccd1592cd73464f"} err="failed to get container status \"1d888c7c1b675a42d0ebd92a6a9b3c76ef3bc60f366e8d340ccd1592cd73464f\": rpc error: code = NotFound desc = could not find container \"1d888c7c1b675a42d0ebd92a6a9b3c76ef3bc60f366e8d340ccd1592cd73464f\": container with ID starting with 1d888c7c1b675a42d0ebd92a6a9b3c76ef3bc60f366e8d340ccd1592cd73464f not found: ID does not exist" Aug 04 11:06:38.738054 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:38.737953 2580 scope.go:117] "RemoveContainer" containerID="5619a393666e208529b318d6401d8f6ef8eb03cd7e405e9edc9b5fccc466f5f1" Aug 04 11:06:38.738306 ip-10-0-142-113 kubenswrapper[2580]: E0804 11:06:38.738284 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5619a393666e208529b318d6401d8f6ef8eb03cd7e405e9edc9b5fccc466f5f1\": container with ID starting with 5619a393666e208529b318d6401d8f6ef8eb03cd7e405e9edc9b5fccc466f5f1 not found: ID does not exist" containerID="5619a393666e208529b318d6401d8f6ef8eb03cd7e405e9edc9b5fccc466f5f1" Aug 04 11:06:38.738396 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:38.738315 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5619a393666e208529b318d6401d8f6ef8eb03cd7e405e9edc9b5fccc466f5f1"} err="failed to get container status \"5619a393666e208529b318d6401d8f6ef8eb03cd7e405e9edc9b5fccc466f5f1\": rpc error: code = NotFound desc = could not find container \"5619a393666e208529b318d6401d8f6ef8eb03cd7e405e9edc9b5fccc466f5f1\": container with ID starting with 5619a393666e208529b318d6401d8f6ef8eb03cd7e405e9edc9b5fccc466f5f1 not found: ID does not exist" Aug 04 11:06:38.740489 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:38.740468 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd"] Aug 04 11:06:38.743866 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:38.743845 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a18af8fb-openshift-default-6b84484bfb-5skdd"] Aug 04 11:06:39.099985 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:39.099905 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3c6157d-9d6d-4fd1-931d-d9487d43a461" path="/var/lib/kubelet/pods/b3c6157d-9d6d-4fd1-931d-d9487d43a461/volumes" Aug 04 11:06:46.825509 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:46.825476 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:06:53.439393 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:06:53.439342 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:07:02.046114 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:02.046069 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:07:39.196789 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.196755 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx"] Aug 04 11:07:39.197260 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.197092 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af15cace-fad3-44fe-8824-23d57674220d" containerName="main" Aug 04 11:07:39.197260 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.197103 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="af15cace-fad3-44fe-8824-23d57674220d" containerName="main" Aug 04 11:07:39.197260 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.197117 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3c6157d-9d6d-4fd1-931d-d9487d43a461" containerName="istio-proxy" Aug 04 11:07:39.197260 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.197123 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3c6157d-9d6d-4fd1-931d-d9487d43a461" containerName="istio-proxy" Aug 04 11:07:39.197260 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.197132 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3c6157d-9d6d-4fd1-931d-d9487d43a461" containerName="istio-proxy" Aug 04 11:07:39.197260 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.197138 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3c6157d-9d6d-4fd1-931d-d9487d43a461" containerName="istio-proxy" Aug 04 11:07:39.197260 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.197184 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3c6157d-9d6d-4fd1-931d-d9487d43a461" containerName="istio-proxy" Aug 04 11:07:39.197260 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.197191 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="af15cace-fad3-44fe-8824-23d57674220d" containerName="main" Aug 04 11:07:39.197260 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.197200 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3c6157d-9d6d-4fd1-931d-d9487d43a461" containerName="istio-proxy" Aug 04 11:07:39.199229 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.199209 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:39.203617 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.203585 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-78f47f-openshift-default-dockercfg-rpppl\"" Aug 04 11:07:39.214045 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.214015 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx"] Aug 04 11:07:39.296038 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.295983 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-istiod-ca-cert\") pod \"e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:39.296218 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.296049 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-workload-certs\") pod \"e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:39.296218 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.296074 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwq6c\" (UniqueName: \"kubernetes.io/projected/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-kube-api-access-hwq6c\") pod \"e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:39.296218 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.296103 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-credential-socket\") pod \"e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:39.296218 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.296172 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-istio-podinfo\") pod \"e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:39.296410 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.296231 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-workload-socket\") pod \"e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:39.296410 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.296281 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-istio-envoy\") pod \"e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:39.296410 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.296366 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-istio-data\") pod \"e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:39.296410 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.296388 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-istio-token\") pod \"e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:39.397022 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.396976 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-workload-certs\") pod \"e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:39.397022 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.397024 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwq6c\" (UniqueName: \"kubernetes.io/projected/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-kube-api-access-hwq6c\") pod \"e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:39.397280 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.397051 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-credential-socket\") pod \"e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:39.397280 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.397077 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-istio-podinfo\") pod \"e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:39.397280 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.397110 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-workload-socket\") pod \"e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:39.397280 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.397140 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-istio-envoy\") pod \"e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:39.397280 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.397221 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-istio-data\") pod \"e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:39.397280 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.397244 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-istio-token\") pod \"e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:39.397641 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.397307 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-istiod-ca-cert\") pod \"e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:39.397641 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.397376 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-workload-certs\") pod \"e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:39.397641 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.397624 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-workload-socket\") pod \"e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:39.397808 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.397658 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-istio-data\") pod \"e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:39.397808 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.397749 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-credential-socket\") pod \"e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:39.398081 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.398060 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-istiod-ca-cert\") pod \"e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:39.400040 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.400017 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-istio-envoy\") pod \"e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:39.400230 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.400211 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-istio-podinfo\") pod \"e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:39.405996 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.405968 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-istio-token\") pod \"e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:39.406156 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.406132 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwq6c\" (UniqueName: \"kubernetes.io/projected/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-kube-api-access-hwq6c\") pod \"e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:39.509286 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.509184 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:39.655245 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.655218 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx"] Aug 04 11:07:39.657888 ip-10-0-142-113 kubenswrapper[2580]: W0804 11:07:39.657861 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b2f7dbf_fbbe_4693_a928_2a1059cd18a0.slice/crio-c0dee0b99cd9b3534dad6e42a1d283edb2820df41fb824ad0e62f65ff34eef53 WatchSource:0}: Error finding container c0dee0b99cd9b3534dad6e42a1d283edb2820df41fb824ad0e62f65ff34eef53: Status 404 returned error can't find the container with id c0dee0b99cd9b3534dad6e42a1d283edb2820df41fb824ad0e62f65ff34eef53 Aug 04 11:07:39.660300 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.660268 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:07:39.660388 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.660336 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:07:39.660388 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:39.660364 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:07:40.002960 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:40.002924 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" event={"ID":"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0","Type":"ContainerStarted","Data":"84d942ded81c51a709443a079e99dc860babf118385506b8fdca22ff7bebf02f"} Aug 04 11:07:40.003174 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:40.002974 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" event={"ID":"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0","Type":"ContainerStarted","Data":"c0dee0b99cd9b3534dad6e42a1d283edb2820df41fb824ad0e62f65ff34eef53"} Aug 04 11:07:40.026889 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:40.026841 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" podStartSLOduration=1.026825654 podStartE2EDuration="1.026825654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:07:40.025427365 +0000 UTC m=+2399.446314758" watchObservedRunningTime="2026-08-04 11:07:40.026825654 +0000 UTC m=+2399.447713064" Aug 04 11:07:40.509425 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:40.509376 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:40.514534 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:40.514509 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:41.007487 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:41.007453 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:41.008534 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:41.008518 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:41.188487 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:41.188461 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tb4zk_c5922147-cb42-4f62-9880-8deaa51a4b0f/ovn-acl-logging/0.log" Aug 04 11:07:41.194403 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:41.194382 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tb4zk_c5922147-cb42-4f62-9880-8deaa51a4b0f/ovn-acl-logging/0.log" Aug 04 11:07:46.136176 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:46.136127 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:07:56.009445 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:56.009395 2580 patch_prober.go:28] interesting pod/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Aug 04 11:07:56.009924 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:56.009467 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" podUID="2b2f7dbf-fbbe-4693-a928-2a1059cd18a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Aug 04 11:07:56.075819 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:56.075785 2580 generic.go:356] "Generic (PLEG): container finished" podID="2b2f7dbf-fbbe-4693-a928-2a1059cd18a0" containerID="84d942ded81c51a709443a079e99dc860babf118385506b8fdca22ff7bebf02f" exitCode=0 Aug 04 11:07:56.076006 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:56.075862 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" event={"ID":"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0","Type":"ContainerDied","Data":"84d942ded81c51a709443a079e99dc860babf118385506b8fdca22ff7bebf02f"} Aug 04 11:07:56.076289 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:56.076273 2580 scope.go:117] "RemoveContainer" containerID="84d942ded81c51a709443a079e99dc860babf118385506b8fdca22ff7bebf02f" Aug 04 11:07:56.077439 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:56.077402 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:07:56.077508 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:56.077489 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:07:56.077546 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:56.077522 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:07:57.092240 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:57.092202 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" event={"ID":"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0","Type":"ContainerStarted","Data":"5d48afd5fd5f9434a7cef5773b6bd798b42f6fd51a65049ef57ac87fe8ee42e2"} Aug 04 11:07:57.509814 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:57.509775 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:07:57.510018 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:57.509995 2580 patch_prober.go:28] interesting pod/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Aug 04 11:07:57.510084 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:57.510044 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" podUID="2b2f7dbf-fbbe-4693-a928-2a1059cd18a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Aug 04 11:07:58.509538 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:58.509505 2580 patch_prober.go:28] interesting pod/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Aug 04 11:07:58.510019 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:58.509591 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" podUID="2b2f7dbf-fbbe-4693-a928-2a1059cd18a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Aug 04 11:07:59.510184 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:59.510142 2580 patch_prober.go:28] interesting pod/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Aug 04 11:07:59.510602 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:07:59.510211 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" podUID="2b2f7dbf-fbbe-4693-a928-2a1059cd18a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Aug 04 11:08:00.509677 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:00.509633 2580 patch_prober.go:28] interesting pod/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Aug 04 11:08:00.509868 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:00.509694 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" podUID="2b2f7dbf-fbbe-4693-a928-2a1059cd18a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Aug 04 11:08:01.509685 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:01.509597 2580 patch_prober.go:28] interesting pod/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Aug 04 11:08:01.509685 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:01.509663 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" podUID="2b2f7dbf-fbbe-4693-a928-2a1059cd18a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Aug 04 11:08:02.513822 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:02.513788 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:08:02.514188 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:02.514069 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:08:02.514745 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:02.514729 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:08:07.616514 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:07.616468 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg"] Aug 04 11:08:07.619220 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:07.619200 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:07.622107 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:07.622085 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1c7c26-openshift-default-dockercfg-r7ctk\"" Aug 04 11:08:07.634146 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:07.634114 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg"] Aug 04 11:08:07.661819 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:07.661774 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9908a34-d679-49a6-9c97-0a2ebf736e1b-credential-socket\") pod \"e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:07.661819 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:07.661817 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9908a34-d679-49a6-9c97-0a2ebf736e1b-istio-envoy\") pod \"e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:07.662047 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:07.661846 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9908a34-d679-49a6-9c97-0a2ebf736e1b-workload-certs\") pod \"e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:07.662047 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:07.661965 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9908a34-d679-49a6-9c97-0a2ebf736e1b-istio-data\") pod \"e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:07.662047 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:07.661993 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9908a34-d679-49a6-9c97-0a2ebf736e1b-istio-podinfo\") pod \"e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:07.662047 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:07.662013 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9908a34-d679-49a6-9c97-0a2ebf736e1b-istiod-ca-cert\") pod \"e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:07.662183 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:07.662076 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhg2x\" (UniqueName: \"kubernetes.io/projected/b9908a34-d679-49a6-9c97-0a2ebf736e1b-kube-api-access-mhg2x\") pod \"e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:07.662183 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:07.662115 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9908a34-d679-49a6-9c97-0a2ebf736e1b-workload-socket\") pod \"e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:07.662183 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:07.662132 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9908a34-d679-49a6-9c97-0a2ebf736e1b-istio-token\") pod \"e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:07.762690 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:07.762654 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9908a34-d679-49a6-9c97-0a2ebf736e1b-workload-socket\") pod \"e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:07.762690 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:07.762692 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9908a34-d679-49a6-9c97-0a2ebf736e1b-istio-token\") pod \"e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:07.762940 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:07.762737 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9908a34-d679-49a6-9c97-0a2ebf736e1b-credential-socket\") pod \"e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:07.762940 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:07.762850 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9908a34-d679-49a6-9c97-0a2ebf736e1b-istio-envoy\") pod \"e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:07.762940 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:07.762904 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9908a34-d679-49a6-9c97-0a2ebf736e1b-workload-certs\") pod \"e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:07.763090 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:07.762986 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9908a34-d679-49a6-9c97-0a2ebf736e1b-istio-data\") pod \"e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:07.763090 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:07.763014 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9908a34-d679-49a6-9c97-0a2ebf736e1b-istio-podinfo\") pod \"e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:07.763090 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:07.763044 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9908a34-d679-49a6-9c97-0a2ebf736e1b-istiod-ca-cert\") pod \"e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:07.763239 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:07.763102 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhg2x\" (UniqueName: \"kubernetes.io/projected/b9908a34-d679-49a6-9c97-0a2ebf736e1b-kube-api-access-mhg2x\") pod \"e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:07.763239 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:07.763147 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9908a34-d679-49a6-9c97-0a2ebf736e1b-credential-socket\") pod \"e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:07.763239 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:07.763197 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9908a34-d679-49a6-9c97-0a2ebf736e1b-workload-socket\") pod \"e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:07.763385 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:07.763269 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9908a34-d679-49a6-9c97-0a2ebf736e1b-workload-certs\") pod \"e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:07.763385 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:07.763355 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9908a34-d679-49a6-9c97-0a2ebf736e1b-istio-data\") pod \"e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:07.763819 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:07.763801 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9908a34-d679-49a6-9c97-0a2ebf736e1b-istiod-ca-cert\") pod \"e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:07.765452 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:07.765428 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9908a34-d679-49a6-9c97-0a2ebf736e1b-istio-envoy\") pod \"e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:07.765656 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:07.765635 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9908a34-d679-49a6-9c97-0a2ebf736e1b-istio-podinfo\") pod \"e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:07.775240 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:07.775203 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9908a34-d679-49a6-9c97-0a2ebf736e1b-istio-token\") pod \"e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:07.775420 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:07.775380 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhg2x\" (UniqueName: \"kubernetes.io/projected/b9908a34-d679-49a6-9c97-0a2ebf736e1b-kube-api-access-mhg2x\") pod \"e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:07.930006 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:07.929906 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:08.072275 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:08.072245 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg"] Aug 04 11:08:08.074981 ip-10-0-142-113 kubenswrapper[2580]: W0804 11:08:08.074956 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9908a34_d679_49a6_9c97_0a2ebf736e1b.slice/crio-d002f68e4073e99946d79cf781e2db697d44a3581e534ded65906b5ff8716f3e WatchSource:0}: Error finding container d002f68e4073e99946d79cf781e2db697d44a3581e534ded65906b5ff8716f3e: Status 404 returned error can't find the container with id d002f68e4073e99946d79cf781e2db697d44a3581e534ded65906b5ff8716f3e Aug 04 11:08:08.077981 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:08.077949 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:08:08.078063 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:08.078025 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:08:08.078063 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:08.078054 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:08:08.146841 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:08.146812 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" event={"ID":"b9908a34-d679-49a6-9c97-0a2ebf736e1b","Type":"ContainerStarted","Data":"d002f68e4073e99946d79cf781e2db697d44a3581e534ded65906b5ff8716f3e"} Aug 04 11:08:09.152815 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:09.152779 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" event={"ID":"b9908a34-d679-49a6-9c97-0a2ebf736e1b","Type":"ContainerStarted","Data":"c2e20ecc8da91eef7bf15a4ca3ffa3cf11e5fb983e563295f976ff09b87f2088"} Aug 04 11:08:09.177501 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:09.177452 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" podStartSLOduration=2.177435265 podStartE2EDuration="2.177435265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:08:09.175723024 +0000 UTC m=+2428.596610436" watchObservedRunningTime="2026-08-04 11:08:09.177435265 +0000 UTC m=+2428.598322654" Aug 04 11:08:09.930975 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:09.930936 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:09.935934 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:09.935909 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:10.158667 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:10.158634 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:10.159602 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:10.159583 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:14.830916 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:14.830877 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:08:25.160086 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:25.160045 2580 patch_prober.go:28] interesting pod/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Aug 04 11:08:25.160497 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:25.160112 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" podUID="b9908a34-d679-49a6-9c97-0a2ebf736e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Aug 04 11:08:25.236183 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:25.236146 2580 generic.go:356] "Generic (PLEG): container finished" podID="b9908a34-d679-49a6-9c97-0a2ebf736e1b" containerID="c2e20ecc8da91eef7bf15a4ca3ffa3cf11e5fb983e563295f976ff09b87f2088" exitCode=0 Aug 04 11:08:25.236353 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:25.236222 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" event={"ID":"b9908a34-d679-49a6-9c97-0a2ebf736e1b","Type":"ContainerDied","Data":"c2e20ecc8da91eef7bf15a4ca3ffa3cf11e5fb983e563295f976ff09b87f2088"} Aug 04 11:08:25.236679 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:25.236659 2580 scope.go:117] "RemoveContainer" containerID="c2e20ecc8da91eef7bf15a4ca3ffa3cf11e5fb983e563295f976ff09b87f2088" Aug 04 11:08:25.237752 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:25.237719 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:08:25.237837 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:25.237821 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:08:25.237877 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:25.237862 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:08:26.242777 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:26.242742 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" event={"ID":"b9908a34-d679-49a6-9c97-0a2ebf736e1b","Type":"ContainerStarted","Data":"898668c7d4e1ce22dc5daaa1681925013b662defb2c8f371541d6fed6a62d3a0"} Aug 04 11:08:26.930163 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:26.930116 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:26.930364 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:26.930342 2580 patch_prober.go:28] interesting pod/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Aug 04 11:08:26.930414 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:26.930393 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" podUID="b9908a34-d679-49a6-9c97-0a2ebf736e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Aug 04 11:08:27.931033 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:27.930983 2580 patch_prober.go:28] interesting pod/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Aug 04 11:08:27.931435 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:27.931042 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" podUID="b9908a34-d679-49a6-9c97-0a2ebf736e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Aug 04 11:08:28.931091 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:28.931054 2580 patch_prober.go:28] interesting pod/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Aug 04 11:08:28.931507 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:28.931122 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" podUID="b9908a34-d679-49a6-9c97-0a2ebf736e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Aug 04 11:08:29.930896 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:29.930860 2580 patch_prober.go:28] interesting pod/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Aug 04 11:08:29.931079 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:29.930931 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" podUID="b9908a34-d679-49a6-9c97-0a2ebf736e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Aug 04 11:08:30.930861 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:30.930820 2580 patch_prober.go:28] interesting pod/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Aug 04 11:08:30.931253 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:30.930881 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" podUID="b9908a34-d679-49a6-9c97-0a2ebf736e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Aug 04 11:08:31.934484 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:31.934449 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:31.934987 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:31.934835 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:31.935627 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:31.935609 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:08:47.535470 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:47.535424 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8"] Aug 04 11:08:47.538760 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:47.538741 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8" Aug 04 11:08:47.541674 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:47.541648 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-78f47f\"/\"openshift-service-ca.crt\"" Aug 04 11:08:47.543117 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:47.543092 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-78f47f\"/\"default-dockercfg-74qfj\"" Aug 04 11:08:47.543251 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:47.543116 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-78f47f\"/\"kube-root-ca.crt\"" Aug 04 11:08:47.543251 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:47.543100 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-78f47f\"/\"test-model-78f47f-kserve-self-signed-certs\"" Aug 04 11:08:47.549817 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:47.549794 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8"] Aug 04 11:08:47.725715 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:47.725678 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-home\") pod \"test-model-78f47f-kserve-6b7cc6674f-hjvw8\" (UID: \"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929\") " pod="ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8" Aug 04 11:08:47.726029 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:47.725996 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-model-cache\") pod \"test-model-78f47f-kserve-6b7cc6674f-hjvw8\" (UID: \"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929\") " pod="ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8" Aug 04 11:08:47.726178 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:47.726105 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-dshm\") pod \"test-model-78f47f-kserve-6b7cc6674f-hjvw8\" (UID: \"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929\") " pod="ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8" Aug 04 11:08:47.726178 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:47.726132 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-tmp-dir\") pod \"test-model-78f47f-kserve-6b7cc6674f-hjvw8\" (UID: \"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929\") " pod="ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8" Aug 04 11:08:47.726301 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:47.726184 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-tls-certs\") pod \"test-model-78f47f-kserve-6b7cc6674f-hjvw8\" (UID: \"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929\") " pod="ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8" Aug 04 11:08:47.726301 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:47.726230 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skk9g\" (UniqueName: \"kubernetes.io/projected/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-kube-api-access-skk9g\") pod \"test-model-78f47f-kserve-6b7cc6674f-hjvw8\" (UID: \"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929\") " pod="ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8" Aug 04 11:08:47.827110 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:47.827005 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-home\") pod \"test-model-78f47f-kserve-6b7cc6674f-hjvw8\" (UID: \"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929\") " pod="ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8" Aug 04 11:08:47.827110 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:47.827050 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-model-cache\") pod \"test-model-78f47f-kserve-6b7cc6674f-hjvw8\" (UID: \"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929\") " pod="ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8" Aug 04 11:08:47.827350 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:47.827115 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-dshm\") pod \"test-model-78f47f-kserve-6b7cc6674f-hjvw8\" (UID: \"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929\") " pod="ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8" Aug 04 11:08:47.827350 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:47.827138 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-tmp-dir\") pod \"test-model-78f47f-kserve-6b7cc6674f-hjvw8\" (UID: \"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929\") " pod="ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8" Aug 04 11:08:47.827350 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:47.827181 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-tls-certs\") pod \"test-model-78f47f-kserve-6b7cc6674f-hjvw8\" (UID: \"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929\") " pod="ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8" Aug 04 11:08:47.827350 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:47.827221 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skk9g\" (UniqueName: \"kubernetes.io/projected/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-kube-api-access-skk9g\") pod \"test-model-78f47f-kserve-6b7cc6674f-hjvw8\" (UID: \"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929\") " pod="ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8" Aug 04 11:08:47.827549 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:47.827485 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-home\") pod \"test-model-78f47f-kserve-6b7cc6674f-hjvw8\" (UID: \"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929\") " pod="ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8" Aug 04 11:08:47.827549 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:47.827510 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-tmp-dir\") pod \"test-model-78f47f-kserve-6b7cc6674f-hjvw8\" (UID: \"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929\") " pod="ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8" Aug 04 11:08:47.827676 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:47.827592 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-model-cache\") pod \"test-model-78f47f-kserve-6b7cc6674f-hjvw8\" (UID: \"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929\") " pod="ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8" Aug 04 11:08:47.829627 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:47.829603 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-dshm\") pod \"test-model-78f47f-kserve-6b7cc6674f-hjvw8\" (UID: \"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929\") " pod="ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8" Aug 04 11:08:47.829773 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:47.829756 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-tls-certs\") pod \"test-model-78f47f-kserve-6b7cc6674f-hjvw8\" (UID: \"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929\") " pod="ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8" Aug 04 11:08:47.836130 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:47.836102 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skk9g\" (UniqueName: \"kubernetes.io/projected/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-kube-api-access-skk9g\") pod \"test-model-78f47f-kserve-6b7cc6674f-hjvw8\" (UID: \"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929\") " pod="ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8" Aug 04 11:08:47.849141 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:47.849102 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8" Aug 04 11:08:47.993214 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:47.993177 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8"] Aug 04 11:08:47.997779 ip-10-0-142-113 kubenswrapper[2580]: W0804 11:08:47.997747 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3bf8a48_78a9_4ca5_b30e_cfcd62f9a929.slice/crio-877a575c7489e8b4c56d16bc25c30d2afa0871b9242c045edd2412a2059edf84 WatchSource:0}: Error finding container 877a575c7489e8b4c56d16bc25c30d2afa0871b9242c045edd2412a2059edf84: Status 404 returned error can't find the container with id 877a575c7489e8b4c56d16bc25c30d2afa0871b9242c045edd2412a2059edf84 Aug 04 11:08:48.343689 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:48.343654 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8" event={"ID":"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929","Type":"ContainerStarted","Data":"fbb153fee83947fb5b7b8a67f29f5adb1944f376e8c17a5f138693720ad7e705"} Aug 04 11:08:48.343689 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:48.343697 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8" event={"ID":"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929","Type":"ContainerStarted","Data":"877a575c7489e8b4c56d16bc25c30d2afa0871b9242c045edd2412a2059edf84"} Aug 04 11:08:48.364148 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:48.364091 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8" podStartSLOduration=1.364072569 podStartE2EDuration="1.364072569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:08:48.362769185 +0000 UTC m=+2467.783656595" watchObservedRunningTime="2026-08-04 11:08:48.364072569 +0000 UTC m=+2467.784959979" Aug 04 11:08:49.936624 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:49.936588 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:08:57.850200 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:57.850162 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8" Aug 04 11:08:57.850677 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:57.850221 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8" Aug 04 11:08:57.863152 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:57.863127 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8" Aug 04 11:08:58.401571 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:58.401528 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8" Aug 04 11:08:59.528789 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:08:59.528708 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:09:16.248260 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:16.248222 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9"] Aug 04 11:09:16.251908 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:16.251885 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9" Aug 04 11:09:16.256264 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:16.256232 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1c7c26\"/\"kube-root-ca.crt\"" Aug 04 11:09:16.256389 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:16.256358 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1c7c26\"/\"test-model-1c7c26-kserve-self-signed-certs\"" Aug 04 11:09:16.256462 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:16.256393 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1c7c26\"/\"openshift-service-ca.crt\"" Aug 04 11:09:16.256462 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:16.256452 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1c7c26\"/\"default-dockercfg-6jw9z\"" Aug 04 11:09:16.262827 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:16.262802 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9"] Aug 04 11:09:16.380007 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:16.379971 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8v42\" (UniqueName: \"kubernetes.io/projected/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-kube-api-access-s8v42\") pod \"test-model-1c7c26-kserve-5cf4c877cc-5lnt9\" (UID: \"d2bc4229-6b77-4f38-a668-5efc4fc5fffd\") " pod="ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9" Aug 04 11:09:16.380168 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:16.380021 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-home\") pod \"test-model-1c7c26-kserve-5cf4c877cc-5lnt9\" (UID: \"d2bc4229-6b77-4f38-a668-5efc4fc5fffd\") " pod="ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9" Aug 04 11:09:16.380168 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:16.380058 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-dshm\") pod \"test-model-1c7c26-kserve-5cf4c877cc-5lnt9\" (UID: \"d2bc4229-6b77-4f38-a668-5efc4fc5fffd\") " pod="ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9" Aug 04 11:09:16.380168 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:16.380147 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-tls-certs\") pod \"test-model-1c7c26-kserve-5cf4c877cc-5lnt9\" (UID: \"d2bc4229-6b77-4f38-a668-5efc4fc5fffd\") " pod="ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9" Aug 04 11:09:16.380275 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:16.380186 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-model-cache\") pod \"test-model-1c7c26-kserve-5cf4c877cc-5lnt9\" (UID: \"d2bc4229-6b77-4f38-a668-5efc4fc5fffd\") " pod="ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9" Aug 04 11:09:16.380275 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:16.380252 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-tmp-dir\") pod \"test-model-1c7c26-kserve-5cf4c877cc-5lnt9\" (UID: \"d2bc4229-6b77-4f38-a668-5efc4fc5fffd\") " pod="ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9" Aug 04 11:09:16.481252 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:16.481216 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-home\") pod \"test-model-1c7c26-kserve-5cf4c877cc-5lnt9\" (UID: \"d2bc4229-6b77-4f38-a668-5efc4fc5fffd\") " pod="ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9" Aug 04 11:09:16.481252 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:16.481254 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-dshm\") pod \"test-model-1c7c26-kserve-5cf4c877cc-5lnt9\" (UID: \"d2bc4229-6b77-4f38-a668-5efc4fc5fffd\") " pod="ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9" Aug 04 11:09:16.481515 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:16.481281 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-tls-certs\") pod \"test-model-1c7c26-kserve-5cf4c877cc-5lnt9\" (UID: \"d2bc4229-6b77-4f38-a668-5efc4fc5fffd\") " pod="ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9" Aug 04 11:09:16.481515 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:16.481298 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-model-cache\") pod \"test-model-1c7c26-kserve-5cf4c877cc-5lnt9\" (UID: \"d2bc4229-6b77-4f38-a668-5efc4fc5fffd\") " pod="ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9" Aug 04 11:09:16.481515 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:16.481319 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-tmp-dir\") pod \"test-model-1c7c26-kserve-5cf4c877cc-5lnt9\" (UID: \"d2bc4229-6b77-4f38-a668-5efc4fc5fffd\") " pod="ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9" Aug 04 11:09:16.481515 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:16.481363 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8v42\" (UniqueName: \"kubernetes.io/projected/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-kube-api-access-s8v42\") pod \"test-model-1c7c26-kserve-5cf4c877cc-5lnt9\" (UID: \"d2bc4229-6b77-4f38-a668-5efc4fc5fffd\") " pod="ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9" Aug 04 11:09:16.481757 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:16.481714 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-home\") pod \"test-model-1c7c26-kserve-5cf4c877cc-5lnt9\" (UID: \"d2bc4229-6b77-4f38-a668-5efc4fc5fffd\") " pod="ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9" Aug 04 11:09:16.481757 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:16.481743 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-model-cache\") pod \"test-model-1c7c26-kserve-5cf4c877cc-5lnt9\" (UID: \"d2bc4229-6b77-4f38-a668-5efc4fc5fffd\") " pod="ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9" Aug 04 11:09:16.481862 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:16.481771 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-tmp-dir\") pod \"test-model-1c7c26-kserve-5cf4c877cc-5lnt9\" (UID: \"d2bc4229-6b77-4f38-a668-5efc4fc5fffd\") " pod="ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9" Aug 04 11:09:16.483799 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:16.483778 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-dshm\") pod \"test-model-1c7c26-kserve-5cf4c877cc-5lnt9\" (UID: \"d2bc4229-6b77-4f38-a668-5efc4fc5fffd\") " pod="ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9" Aug 04 11:09:16.483913 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:16.483881 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-tls-certs\") pod \"test-model-1c7c26-kserve-5cf4c877cc-5lnt9\" (UID: \"d2bc4229-6b77-4f38-a668-5efc4fc5fffd\") " pod="ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9" Aug 04 11:09:16.491144 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:16.491117 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8v42\" (UniqueName: \"kubernetes.io/projected/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-kube-api-access-s8v42\") pod \"test-model-1c7c26-kserve-5cf4c877cc-5lnt9\" (UID: \"d2bc4229-6b77-4f38-a668-5efc4fc5fffd\") " pod="ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9" Aug 04 11:09:16.562914 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:16.562820 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9" Aug 04 11:09:16.704708 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:16.704676 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9"] Aug 04 11:09:16.707641 ip-10-0-142-113 kubenswrapper[2580]: W0804 11:09:16.707612 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2bc4229_6b77_4f38_a668_5efc4fc5fffd.slice/crio-bf921cc7613227279f621b334f24418dcd610fb34902dc34373cb02ee03bd67c WatchSource:0}: Error finding container bf921cc7613227279f621b334f24418dcd610fb34902dc34373cb02ee03bd67c: Status 404 returned error can't find the container with id bf921cc7613227279f621b334f24418dcd610fb34902dc34373cb02ee03bd67c Aug 04 11:09:17.492766 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:17.492724 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9" event={"ID":"d2bc4229-6b77-4f38-a668-5efc4fc5fffd","Type":"ContainerStarted","Data":"318e5488c4787783a4b55e8525bc258eb0b3f7eaf28c34a3d434b775efd5576f"} Aug 04 11:09:17.493305 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:17.492784 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9" event={"ID":"d2bc4229-6b77-4f38-a668-5efc4fc5fffd","Type":"ContainerStarted","Data":"bf921cc7613227279f621b334f24418dcd610fb34902dc34373cb02ee03bd67c"} Aug 04 11:09:17.511764 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:17.511706 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9" podStartSLOduration=1.511690319 podStartE2EDuration="1.511690319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:09:17.511243333 +0000 UTC m=+2496.932130744" watchObservedRunningTime="2026-08-04 11:09:17.511690319 +0000 UTC m=+2496.932577712" Aug 04 11:09:19.438136 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:19.438105 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:09:26.563713 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:26.563669 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9" Aug 04 11:09:26.563713 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:26.563726 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9" Aug 04 11:09:26.576500 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:26.576471 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9" Aug 04 11:09:27.549895 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:27.549865 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9" Aug 04 11:09:28.039699 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:09:28.039662 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:10:01.038060 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:01.038012 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8"] Aug 04 11:10:01.926225 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:01.926193 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9"] Aug 04 11:10:02.642047 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:02.642002 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:10:08.933392 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:08.933354 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:10:16.057399 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:16.057357 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8" podUID="c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929" containerName="main" containerID="cri-o://fbb153fee83947fb5b7b8a67f29f5adb1944f376e8c17a5f138693720ad7e705" gracePeriod=45 Aug 04 11:10:16.306088 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:16.306061 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8" Aug 04 11:10:16.435007 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:16.434966 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-tmp-dir\") pod \"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929\" (UID: \"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929\") " Aug 04 11:10:16.435208 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:16.435045 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-dshm\") pod \"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929\" (UID: \"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929\") " Aug 04 11:10:16.435208 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:16.435076 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-home\") pod \"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929\" (UID: \"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929\") " Aug 04 11:10:16.435208 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:16.435092 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-model-cache\") pod \"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929\" (UID: \"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929\") " Aug 04 11:10:16.435208 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:16.435133 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skk9g\" (UniqueName: \"kubernetes.io/projected/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-kube-api-access-skk9g\") pod \"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929\" (UID: \"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929\") " Aug 04 11:10:16.435208 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:16.435154 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-tls-certs\") pod \"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929\" (UID: \"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929\") " Aug 04 11:10:16.435478 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:16.435316 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929" (UID: "c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:10:16.435478 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:16.435352 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-model-cache" (OuterVolumeSpecName: "model-cache") pod "c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929" (UID: "c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:10:16.435478 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:16.435371 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-home" (OuterVolumeSpecName: "home") pod "c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929" (UID: "c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:10:16.435661 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:16.435484 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-home\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:10:16.435661 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:16.435502 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-model-cache\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:10:16.435661 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:16.435518 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-tmp-dir\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:10:16.437631 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:16.437603 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-dshm" (OuterVolumeSpecName: "dshm") pod "c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929" (UID: "c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:10:16.437742 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:16.437659 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929" (UID: "c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:10:16.437742 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:16.437682 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-kube-api-access-skk9g" (OuterVolumeSpecName: "kube-api-access-skk9g") pod "c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929" (UID: "c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929"). InnerVolumeSpecName "kube-api-access-skk9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:10:16.536385 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:16.536348 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-dshm\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:10:16.536385 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:16.536379 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skk9g\" (UniqueName: \"kubernetes.io/projected/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-kube-api-access-skk9g\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:10:16.536385 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:16.536389 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929-tls-certs\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:10:16.791909 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:16.791824 2580 generic.go:356] "Generic (PLEG): container finished" podID="c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929" containerID="fbb153fee83947fb5b7b8a67f29f5adb1944f376e8c17a5f138693720ad7e705" exitCode=0 Aug 04 11:10:16.791909 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:16.791870 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8" Aug 04 11:10:16.792093 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:16.791916 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8" event={"ID":"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929","Type":"ContainerDied","Data":"fbb153fee83947fb5b7b8a67f29f5adb1944f376e8c17a5f138693720ad7e705"} Aug 04 11:10:16.792093 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:16.791963 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8" event={"ID":"c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929","Type":"ContainerDied","Data":"877a575c7489e8b4c56d16bc25c30d2afa0871b9242c045edd2412a2059edf84"} Aug 04 11:10:16.792093 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:16.791982 2580 scope.go:117] "RemoveContainer" containerID="fbb153fee83947fb5b7b8a67f29f5adb1944f376e8c17a5f138693720ad7e705" Aug 04 11:10:16.801266 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:16.801250 2580 scope.go:117] "RemoveContainer" containerID="fbb153fee83947fb5b7b8a67f29f5adb1944f376e8c17a5f138693720ad7e705" Aug 04 11:10:16.801611 ip-10-0-142-113 kubenswrapper[2580]: E0804 11:10:16.801547 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbb153fee83947fb5b7b8a67f29f5adb1944f376e8c17a5f138693720ad7e705\": container with ID starting with fbb153fee83947fb5b7b8a67f29f5adb1944f376e8c17a5f138693720ad7e705 not found: ID does not exist" containerID="fbb153fee83947fb5b7b8a67f29f5adb1944f376e8c17a5f138693720ad7e705" Aug 04 11:10:16.801686 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:16.801620 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbb153fee83947fb5b7b8a67f29f5adb1944f376e8c17a5f138693720ad7e705"} err="failed to get container status \"fbb153fee83947fb5b7b8a67f29f5adb1944f376e8c17a5f138693720ad7e705\": rpc error: code = NotFound desc = could not find container \"fbb153fee83947fb5b7b8a67f29f5adb1944f376e8c17a5f138693720ad7e705\": container with ID starting with fbb153fee83947fb5b7b8a67f29f5adb1944f376e8c17a5f138693720ad7e705 not found: ID does not exist" Aug 04 11:10:16.812989 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:16.812962 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8"] Aug 04 11:10:16.815002 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:16.814979 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-78f47f/test-model-78f47f-kserve-6b7cc6674f-hjvw8"] Aug 04 11:10:16.946463 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:16.946423 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9" podUID="d2bc4229-6b77-4f38-a668-5efc4fc5fffd" containerName="main" containerID="cri-o://318e5488c4787783a4b55e8525bc258eb0b3f7eaf28c34a3d434b775efd5576f" gracePeriod=45 Aug 04 11:10:17.101133 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:17.101099 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929" path="/var/lib/kubelet/pods/c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929/volumes" Aug 04 11:10:17.204017 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:17.203993 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9" Aug 04 11:10:17.344892 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:17.344802 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8v42\" (UniqueName: \"kubernetes.io/projected/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-kube-api-access-s8v42\") pod \"d2bc4229-6b77-4f38-a668-5efc4fc5fffd\" (UID: \"d2bc4229-6b77-4f38-a668-5efc4fc5fffd\") " Aug 04 11:10:17.344892 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:17.344847 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-home\") pod \"d2bc4229-6b77-4f38-a668-5efc4fc5fffd\" (UID: \"d2bc4229-6b77-4f38-a668-5efc4fc5fffd\") " Aug 04 11:10:17.344892 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:17.344877 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-model-cache\") pod \"d2bc4229-6b77-4f38-a668-5efc4fc5fffd\" (UID: \"d2bc4229-6b77-4f38-a668-5efc4fc5fffd\") " Aug 04 11:10:17.345188 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:17.344918 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-tmp-dir\") pod \"d2bc4229-6b77-4f38-a668-5efc4fc5fffd\" (UID: \"d2bc4229-6b77-4f38-a668-5efc4fc5fffd\") " Aug 04 11:10:17.345188 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:17.344954 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-dshm\") pod \"d2bc4229-6b77-4f38-a668-5efc4fc5fffd\" (UID: \"d2bc4229-6b77-4f38-a668-5efc4fc5fffd\") " Aug 04 11:10:17.345188 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:17.344979 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-tls-certs\") pod \"d2bc4229-6b77-4f38-a668-5efc4fc5fffd\" (UID: \"d2bc4229-6b77-4f38-a668-5efc4fc5fffd\") " Aug 04 11:10:17.345188 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:17.345129 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-home" (OuterVolumeSpecName: "home") pod "d2bc4229-6b77-4f38-a668-5efc4fc5fffd" (UID: "d2bc4229-6b77-4f38-a668-5efc4fc5fffd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:10:17.345395 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:17.345185 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-model-cache" (OuterVolumeSpecName: "model-cache") pod "d2bc4229-6b77-4f38-a668-5efc4fc5fffd" (UID: "d2bc4229-6b77-4f38-a668-5efc4fc5fffd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:10:17.345395 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:17.345201 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d2bc4229-6b77-4f38-a668-5efc4fc5fffd" (UID: "d2bc4229-6b77-4f38-a668-5efc4fc5fffd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:10:17.345395 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:17.345303 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-home\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:10:17.345395 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:17.345322 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-model-cache\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:10:17.345395 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:17.345337 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-tmp-dir\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:10:17.347292 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:17.347271 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-dshm" (OuterVolumeSpecName: "dshm") pod "d2bc4229-6b77-4f38-a668-5efc4fc5fffd" (UID: "d2bc4229-6b77-4f38-a668-5efc4fc5fffd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:10:17.347405 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:17.347332 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-kube-api-access-s8v42" (OuterVolumeSpecName: "kube-api-access-s8v42") pod "d2bc4229-6b77-4f38-a668-5efc4fc5fffd" (UID: "d2bc4229-6b77-4f38-a668-5efc4fc5fffd"). InnerVolumeSpecName "kube-api-access-s8v42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:10:17.347405 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:17.347391 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d2bc4229-6b77-4f38-a668-5efc4fc5fffd" (UID: "d2bc4229-6b77-4f38-a668-5efc4fc5fffd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:10:17.446515 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:17.446471 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-dshm\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:10:17.446515 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:17.446506 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-tls-certs\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:10:17.446515 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:17.446519 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8v42\" (UniqueName: \"kubernetes.io/projected/d2bc4229-6b77-4f38-a668-5efc4fc5fffd-kube-api-access-s8v42\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:10:17.799109 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:17.799076 2580 generic.go:356] "Generic (PLEG): container finished" podID="d2bc4229-6b77-4f38-a668-5efc4fc5fffd" containerID="318e5488c4787783a4b55e8525bc258eb0b3f7eaf28c34a3d434b775efd5576f" exitCode=0 Aug 04 11:10:17.799306 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:17.799117 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9" event={"ID":"d2bc4229-6b77-4f38-a668-5efc4fc5fffd","Type":"ContainerDied","Data":"318e5488c4787783a4b55e8525bc258eb0b3f7eaf28c34a3d434b775efd5576f"} Aug 04 11:10:17.799306 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:17.799122 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9" Aug 04 11:10:17.799306 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:17.799144 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9" event={"ID":"d2bc4229-6b77-4f38-a668-5efc4fc5fffd","Type":"ContainerDied","Data":"bf921cc7613227279f621b334f24418dcd610fb34902dc34373cb02ee03bd67c"} Aug 04 11:10:17.799306 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:17.799161 2580 scope.go:117] "RemoveContainer" containerID="318e5488c4787783a4b55e8525bc258eb0b3f7eaf28c34a3d434b775efd5576f" Aug 04 11:10:17.808058 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:17.808043 2580 scope.go:117] "RemoveContainer" containerID="318e5488c4787783a4b55e8525bc258eb0b3f7eaf28c34a3d434b775efd5576f" Aug 04 11:10:17.808323 ip-10-0-142-113 kubenswrapper[2580]: E0804 11:10:17.808305 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"318e5488c4787783a4b55e8525bc258eb0b3f7eaf28c34a3d434b775efd5576f\": container with ID starting with 318e5488c4787783a4b55e8525bc258eb0b3f7eaf28c34a3d434b775efd5576f not found: ID does not exist" containerID="318e5488c4787783a4b55e8525bc258eb0b3f7eaf28c34a3d434b775efd5576f" Aug 04 11:10:17.808364 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:17.808334 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"318e5488c4787783a4b55e8525bc258eb0b3f7eaf28c34a3d434b775efd5576f"} err="failed to get container status \"318e5488c4787783a4b55e8525bc258eb0b3f7eaf28c34a3d434b775efd5576f\": rpc error: code = NotFound desc = could not find container \"318e5488c4787783a4b55e8525bc258eb0b3f7eaf28c34a3d434b775efd5576f\": container with ID starting with 318e5488c4787783a4b55e8525bc258eb0b3f7eaf28c34a3d434b775efd5576f not found: ID does not exist" Aug 04 11:10:17.820261 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:17.820232 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9"] Aug 04 11:10:17.822928 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:17.822903 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-1c7c26/test-model-1c7c26-kserve-5cf4c877cc-5lnt9"] Aug 04 11:10:18.731708 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:18.731677 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx"] Aug 04 11:10:18.732114 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:18.731887 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" podUID="2b2f7dbf-fbbe-4693-a928-2a1059cd18a0" containerName="istio-proxy" containerID="cri-o://5d48afd5fd5f9434a7cef5773b6bd798b42f6fd51a65049ef57ac87fe8ee42e2" gracePeriod=30 Aug 04 11:10:19.099931 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:19.099850 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2bc4229-6b77-4f38-a668-5efc4fc5fffd" path="/var/lib/kubelet/pods/d2bc4229-6b77-4f38-a668-5efc4fc5fffd/volumes" Aug 04 11:10:23.981670 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:23.981647 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:10:24.114735 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.114701 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-workload-certs\") pod \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " Aug 04 11:10:24.114929 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.114743 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwq6c\" (UniqueName: \"kubernetes.io/projected/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-kube-api-access-hwq6c\") pod \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " Aug 04 11:10:24.114929 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.114769 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-istio-token\") pod \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " Aug 04 11:10:24.114929 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.114801 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-credential-socket\") pod \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " Aug 04 11:10:24.114929 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.114908 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-workload-socket\") pod \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " Aug 04 11:10:24.115117 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.114948 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-istio-data\") pod \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " Aug 04 11:10:24.115117 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.115001 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-istiod-ca-cert\") pod \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " Aug 04 11:10:24.115117 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.114997 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2b2f7dbf-fbbe-4693-a928-2a1059cd18a0" (UID: "2b2f7dbf-fbbe-4693-a928-2a1059cd18a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:10:24.115117 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.115061 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-istio-podinfo\") pod \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " Aug 04 11:10:24.115117 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.115073 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2b2f7dbf-fbbe-4693-a928-2a1059cd18a0" (UID: "2b2f7dbf-fbbe-4693-a928-2a1059cd18a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:10:24.115117 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.115101 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-istio-envoy\") pod \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\" (UID: \"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0\") " Aug 04 11:10:24.115423 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.115200 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2b2f7dbf-fbbe-4693-a928-2a1059cd18a0" (UID: "2b2f7dbf-fbbe-4693-a928-2a1059cd18a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:10:24.115423 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.115288 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-istio-data" (OuterVolumeSpecName: "istio-data") pod "2b2f7dbf-fbbe-4693-a928-2a1059cd18a0" (UID: "2b2f7dbf-fbbe-4693-a928-2a1059cd18a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:10:24.115423 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.115400 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-credential-socket\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:10:24.115423 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.115419 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-workload-socket\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:10:24.115708 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.115420 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2b2f7dbf-fbbe-4693-a928-2a1059cd18a0" (UID: "2b2f7dbf-fbbe-4693-a928-2a1059cd18a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:10:24.115708 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.115435 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-istio-data\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:10:24.115708 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.115450 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-workload-certs\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:10:24.117118 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.117089 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-kube-api-access-hwq6c" (OuterVolumeSpecName: "kube-api-access-hwq6c") pod "2b2f7dbf-fbbe-4693-a928-2a1059cd18a0" (UID: "2b2f7dbf-fbbe-4693-a928-2a1059cd18a0"). InnerVolumeSpecName "kube-api-access-hwq6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:10:24.117436 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.117406 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2b2f7dbf-fbbe-4693-a928-2a1059cd18a0" (UID: "2b2f7dbf-fbbe-4693-a928-2a1059cd18a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:10:24.117688 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.117665 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2b2f7dbf-fbbe-4693-a928-2a1059cd18a0" (UID: "2b2f7dbf-fbbe-4693-a928-2a1059cd18a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 11:10:24.117788 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.117700 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-istio-token" (OuterVolumeSpecName: "istio-token") pod "2b2f7dbf-fbbe-4693-a928-2a1059cd18a0" (UID: "2b2f7dbf-fbbe-4693-a928-2a1059cd18a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:10:24.216405 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.216294 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwq6c\" (UniqueName: \"kubernetes.io/projected/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-kube-api-access-hwq6c\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:10:24.216405 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.216337 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-istio-token\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:10:24.216405 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.216351 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-istiod-ca-cert\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:10:24.216405 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.216365 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-istio-podinfo\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:10:24.216405 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.216378 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0-istio-envoy\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:10:24.836517 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.836480 2580 generic.go:356] "Generic (PLEG): container finished" podID="2b2f7dbf-fbbe-4693-a928-2a1059cd18a0" containerID="5d48afd5fd5f9434a7cef5773b6bd798b42f6fd51a65049ef57ac87fe8ee42e2" exitCode=0 Aug 04 11:10:24.836717 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.836528 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" Aug 04 11:10:24.836717 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.836585 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" event={"ID":"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0","Type":"ContainerDied","Data":"5d48afd5fd5f9434a7cef5773b6bd798b42f6fd51a65049ef57ac87fe8ee42e2"} Aug 04 11:10:24.836717 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.836645 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx" event={"ID":"2b2f7dbf-fbbe-4693-a928-2a1059cd18a0","Type":"ContainerDied","Data":"c0dee0b99cd9b3534dad6e42a1d283edb2820df41fb824ad0e62f65ff34eef53"} Aug 04 11:10:24.836717 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.836675 2580 scope.go:117] "RemoveContainer" containerID="5d48afd5fd5f9434a7cef5773b6bd798b42f6fd51a65049ef57ac87fe8ee42e2" Aug 04 11:10:24.850298 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.850197 2580 scope.go:117] "RemoveContainer" containerID="84d942ded81c51a709443a079e99dc860babf118385506b8fdca22ff7bebf02f" Aug 04 11:10:24.858532 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.858511 2580 scope.go:117] "RemoveContainer" containerID="5d48afd5fd5f9434a7cef5773b6bd798b42f6fd51a65049ef57ac87fe8ee42e2" Aug 04 11:10:24.858861 ip-10-0-142-113 kubenswrapper[2580]: E0804 11:10:24.858840 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d48afd5fd5f9434a7cef5773b6bd798b42f6fd51a65049ef57ac87fe8ee42e2\": container with ID starting with 5d48afd5fd5f9434a7cef5773b6bd798b42f6fd51a65049ef57ac87fe8ee42e2 not found: ID does not exist" containerID="5d48afd5fd5f9434a7cef5773b6bd798b42f6fd51a65049ef57ac87fe8ee42e2" Aug 04 11:10:24.858912 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.858871 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d48afd5fd5f9434a7cef5773b6bd798b42f6fd51a65049ef57ac87fe8ee42e2"} err="failed to get container status \"5d48afd5fd5f9434a7cef5773b6bd798b42f6fd51a65049ef57ac87fe8ee42e2\": rpc error: code = NotFound desc = could not find container \"5d48afd5fd5f9434a7cef5773b6bd798b42f6fd51a65049ef57ac87fe8ee42e2\": container with ID starting with 5d48afd5fd5f9434a7cef5773b6bd798b42f6fd51a65049ef57ac87fe8ee42e2 not found: ID does not exist" Aug 04 11:10:24.858912 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.858891 2580 scope.go:117] "RemoveContainer" containerID="84d942ded81c51a709443a079e99dc860babf118385506b8fdca22ff7bebf02f" Aug 04 11:10:24.859160 ip-10-0-142-113 kubenswrapper[2580]: E0804 11:10:24.859142 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84d942ded81c51a709443a079e99dc860babf118385506b8fdca22ff7bebf02f\": container with ID starting with 84d942ded81c51a709443a079e99dc860babf118385506b8fdca22ff7bebf02f not found: ID does not exist" containerID="84d942ded81c51a709443a079e99dc860babf118385506b8fdca22ff7bebf02f" Aug 04 11:10:24.859205 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.859167 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84d942ded81c51a709443a079e99dc860babf118385506b8fdca22ff7bebf02f"} err="failed to get container status \"84d942ded81c51a709443a079e99dc860babf118385506b8fdca22ff7bebf02f\": rpc error: code = NotFound desc = could not find container \"84d942ded81c51a709443a079e99dc860babf118385506b8fdca22ff7bebf02f\": container with ID starting with 84d942ded81c51a709443a079e99dc860babf118385506b8fdca22ff7bebf02f not found: ID does not exist" Aug 04 11:10:24.863596 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.863552 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx"] Aug 04 11:10:24.868377 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:24.868350 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-78f47f-openshift-default-66c897587d-dvxzx"] Aug 04 11:10:25.099878 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:25.099800 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b2f7dbf-fbbe-4693-a928-2a1059cd18a0" path="/var/lib/kubelet/pods/2b2f7dbf-fbbe-4693-a928-2a1059cd18a0/volumes" Aug 04 11:10:31.627839 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:31.627799 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:10:36.525083 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:36.525050 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg"] Aug 04 11:10:36.525545 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:36.525318 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" podUID="b9908a34-d679-49a6-9c97-0a2ebf736e1b" containerName="istio-proxy" containerID="cri-o://898668c7d4e1ce22dc5daaa1681925013b662defb2c8f371541d6fed6a62d3a0" gracePeriod=30 Aug 04 11:10:37.401456 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.401408 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh"] Aug 04 11:10:37.401875 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.401859 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2bc4229-6b77-4f38-a668-5efc4fc5fffd" containerName="main" Aug 04 11:10:37.401923 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.401878 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2bc4229-6b77-4f38-a668-5efc4fc5fffd" containerName="main" Aug 04 11:10:37.401923 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.401888 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b2f7dbf-fbbe-4693-a928-2a1059cd18a0" containerName="istio-proxy" Aug 04 11:10:37.401923 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.401894 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b2f7dbf-fbbe-4693-a928-2a1059cd18a0" containerName="istio-proxy" Aug 04 11:10:37.401923 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.401901 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b2f7dbf-fbbe-4693-a928-2a1059cd18a0" containerName="istio-proxy" Aug 04 11:10:37.401923 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.401907 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b2f7dbf-fbbe-4693-a928-2a1059cd18a0" containerName="istio-proxy" Aug 04 11:10:37.401923 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.401914 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929" containerName="main" Aug 04 11:10:37.401923 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.401919 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929" containerName="main" Aug 04 11:10:37.402135 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.401980 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b2f7dbf-fbbe-4693-a928-2a1059cd18a0" containerName="istio-proxy" Aug 04 11:10:37.402135 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.401989 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b2f7dbf-fbbe-4693-a928-2a1059cd18a0" containerName="istio-proxy" Aug 04 11:10:37.402135 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.401997 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3bf8a48-78a9-4ca5-b30e-cfcd62f9a929" containerName="main" Aug 04 11:10:37.402135 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.402003 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2bc4229-6b77-4f38-a668-5efc4fc5fffd" containerName="main" Aug 04 11:10:37.406642 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.406619 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:37.409506 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.409485 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-1154a6-openshift-default-dockercfg-7xhxx\"" Aug 04 11:10:37.416603 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.416552 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh"] Aug 04 11:10:37.420277 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.420245 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81cf8e38-9dff-40c5-936c-d48485868cde-istiod-ca-cert\") pod \"e2e-api-a-1154a6-openshift-default-6547464856-xlmqh\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:37.420421 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.420349 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqdxj\" (UniqueName: \"kubernetes.io/projected/81cf8e38-9dff-40c5-936c-d48485868cde-kube-api-access-bqdxj\") pod \"e2e-api-a-1154a6-openshift-default-6547464856-xlmqh\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:37.420552 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.420416 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81cf8e38-9dff-40c5-936c-d48485868cde-workload-socket\") pod \"e2e-api-a-1154a6-openshift-default-6547464856-xlmqh\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:37.420552 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.420465 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81cf8e38-9dff-40c5-936c-d48485868cde-workload-certs\") pod \"e2e-api-a-1154a6-openshift-default-6547464856-xlmqh\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:37.420552 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.420492 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81cf8e38-9dff-40c5-936c-d48485868cde-istio-token\") pod \"e2e-api-a-1154a6-openshift-default-6547464856-xlmqh\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:37.420552 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.420520 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81cf8e38-9dff-40c5-936c-d48485868cde-istio-podinfo\") pod \"e2e-api-a-1154a6-openshift-default-6547464856-xlmqh\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:37.420552 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.420544 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81cf8e38-9dff-40c5-936c-d48485868cde-istio-data\") pod \"e2e-api-a-1154a6-openshift-default-6547464856-xlmqh\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:37.420855 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.420596 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81cf8e38-9dff-40c5-936c-d48485868cde-credential-socket\") pod \"e2e-api-a-1154a6-openshift-default-6547464856-xlmqh\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:37.420855 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.420633 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81cf8e38-9dff-40c5-936c-d48485868cde-istio-envoy\") pod \"e2e-api-a-1154a6-openshift-default-6547464856-xlmqh\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:37.521836 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.521796 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81cf8e38-9dff-40c5-936c-d48485868cde-istio-podinfo\") pod \"e2e-api-a-1154a6-openshift-default-6547464856-xlmqh\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:37.521836 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.521839 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81cf8e38-9dff-40c5-936c-d48485868cde-istio-data\") pod \"e2e-api-a-1154a6-openshift-default-6547464856-xlmqh\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:37.522100 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.521859 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81cf8e38-9dff-40c5-936c-d48485868cde-credential-socket\") pod \"e2e-api-a-1154a6-openshift-default-6547464856-xlmqh\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:37.522100 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.521885 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81cf8e38-9dff-40c5-936c-d48485868cde-istio-envoy\") pod \"e2e-api-a-1154a6-openshift-default-6547464856-xlmqh\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:37.522100 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.521937 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81cf8e38-9dff-40c5-936c-d48485868cde-istiod-ca-cert\") pod \"e2e-api-a-1154a6-openshift-default-6547464856-xlmqh\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:37.522100 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.521985 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqdxj\" (UniqueName: \"kubernetes.io/projected/81cf8e38-9dff-40c5-936c-d48485868cde-kube-api-access-bqdxj\") pod \"e2e-api-a-1154a6-openshift-default-6547464856-xlmqh\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:37.522100 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.522032 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81cf8e38-9dff-40c5-936c-d48485868cde-workload-socket\") pod \"e2e-api-a-1154a6-openshift-default-6547464856-xlmqh\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:37.522100 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.522076 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81cf8e38-9dff-40c5-936c-d48485868cde-workload-certs\") pod \"e2e-api-a-1154a6-openshift-default-6547464856-xlmqh\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:37.522100 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.522099 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81cf8e38-9dff-40c5-936c-d48485868cde-istio-token\") pod \"e2e-api-a-1154a6-openshift-default-6547464856-xlmqh\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:37.522459 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.522284 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81cf8e38-9dff-40c5-936c-d48485868cde-istio-data\") pod \"e2e-api-a-1154a6-openshift-default-6547464856-xlmqh\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:37.522459 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.522343 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81cf8e38-9dff-40c5-936c-d48485868cde-credential-socket\") pod \"e2e-api-a-1154a6-openshift-default-6547464856-xlmqh\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:37.522622 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.522603 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81cf8e38-9dff-40c5-936c-d48485868cde-workload-certs\") pod \"e2e-api-a-1154a6-openshift-default-6547464856-xlmqh\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:37.522684 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.522606 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81cf8e38-9dff-40c5-936c-d48485868cde-workload-socket\") pod \"e2e-api-a-1154a6-openshift-default-6547464856-xlmqh\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:37.522939 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.522916 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81cf8e38-9dff-40c5-936c-d48485868cde-istiod-ca-cert\") pod \"e2e-api-a-1154a6-openshift-default-6547464856-xlmqh\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:37.524374 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.524354 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81cf8e38-9dff-40c5-936c-d48485868cde-istio-envoy\") pod \"e2e-api-a-1154a6-openshift-default-6547464856-xlmqh\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:37.524739 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.524718 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81cf8e38-9dff-40c5-936c-d48485868cde-istio-podinfo\") pod \"e2e-api-a-1154a6-openshift-default-6547464856-xlmqh\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:37.530490 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.530465 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81cf8e38-9dff-40c5-936c-d48485868cde-istio-token\") pod \"e2e-api-a-1154a6-openshift-default-6547464856-xlmqh\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:37.530857 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.530684 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqdxj\" (UniqueName: \"kubernetes.io/projected/81cf8e38-9dff-40c5-936c-d48485868cde-kube-api-access-bqdxj\") pod \"e2e-api-a-1154a6-openshift-default-6547464856-xlmqh\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:37.717626 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.717494 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:37.865170 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.865143 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh"] Aug 04 11:10:37.867217 ip-10-0-142-113 kubenswrapper[2580]: W0804 11:10:37.867186 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81cf8e38_9dff_40c5_936c_d48485868cde.slice/crio-b525208d155007046d77f5d951c5fdcbffaafca9c0a294fa99bd7dc177d59fa1 WatchSource:0}: Error finding container b525208d155007046d77f5d951c5fdcbffaafca9c0a294fa99bd7dc177d59fa1: Status 404 returned error can't find the container with id b525208d155007046d77f5d951c5fdcbffaafca9c0a294fa99bd7dc177d59fa1 Aug 04 11:10:37.869321 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.869285 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:10:37.869436 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.869356 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:10:37.869436 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.869385 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:10:37.903600 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:37.903542 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" event={"ID":"81cf8e38-9dff-40c5-936c-d48485868cde","Type":"ContainerStarted","Data":"b525208d155007046d77f5d951c5fdcbffaafca9c0a294fa99bd7dc177d59fa1"} Aug 04 11:10:38.909794 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:38.909755 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" event={"ID":"81cf8e38-9dff-40c5-936c-d48485868cde","Type":"ContainerStarted","Data":"89697c93f83c8125ca9addb09c6e6bf37a6c47693e551662dba022309aabd332"} Aug 04 11:10:38.933595 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:38.933518 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" podStartSLOduration=1.9335020429999998 podStartE2EDuration="1.933502043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:10:38.931476436 +0000 UTC m=+2578.352363862" watchObservedRunningTime="2026-08-04 11:10:38.933502043 +0000 UTC m=+2578.354389530" Aug 04 11:10:39.718184 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:39.718142 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:39.723204 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:39.723180 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:39.915328 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:39.915292 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:39.916253 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:39.916237 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:41.786221 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.786191 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:10:41.860019 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.859989 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9908a34-d679-49a6-9c97-0a2ebf736e1b-istio-envoy\") pod \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " Aug 04 11:10:41.860019 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.860022 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9908a34-d679-49a6-9c97-0a2ebf736e1b-istio-token\") pod \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " Aug 04 11:10:41.860343 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.860052 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9908a34-d679-49a6-9c97-0a2ebf736e1b-istio-podinfo\") pod \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " Aug 04 11:10:41.860343 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.860073 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhg2x\" (UniqueName: \"kubernetes.io/projected/b9908a34-d679-49a6-9c97-0a2ebf736e1b-kube-api-access-mhg2x\") pod \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " Aug 04 11:10:41.860343 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.860094 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9908a34-d679-49a6-9c97-0a2ebf736e1b-workload-socket\") pod \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " Aug 04 11:10:41.860343 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.860194 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9908a34-d679-49a6-9c97-0a2ebf736e1b-credential-socket\") pod \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " Aug 04 11:10:41.860343 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.860272 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9908a34-d679-49a6-9c97-0a2ebf736e1b-workload-certs\") pod \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " Aug 04 11:10:41.860343 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.860308 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9908a34-d679-49a6-9c97-0a2ebf736e1b-istio-data\") pod \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " Aug 04 11:10:41.860708 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.860350 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9908a34-d679-49a6-9c97-0a2ebf736e1b-istiod-ca-cert\") pod \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\" (UID: \"b9908a34-d679-49a6-9c97-0a2ebf736e1b\") " Aug 04 11:10:41.860708 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.860396 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9908a34-d679-49a6-9c97-0a2ebf736e1b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b9908a34-d679-49a6-9c97-0a2ebf736e1b" (UID: "b9908a34-d679-49a6-9c97-0a2ebf736e1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:10:41.860708 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.860457 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9908a34-d679-49a6-9c97-0a2ebf736e1b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b9908a34-d679-49a6-9c97-0a2ebf736e1b" (UID: "b9908a34-d679-49a6-9c97-0a2ebf736e1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:10:41.860708 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.860596 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9908a34-d679-49a6-9c97-0a2ebf736e1b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b9908a34-d679-49a6-9c97-0a2ebf736e1b" (UID: "b9908a34-d679-49a6-9c97-0a2ebf736e1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:10:41.860708 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.860626 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9908a34-d679-49a6-9c97-0a2ebf736e1b-istio-data" (OuterVolumeSpecName: "istio-data") pod "b9908a34-d679-49a6-9c97-0a2ebf736e1b" (UID: "b9908a34-d679-49a6-9c97-0a2ebf736e1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:10:41.861134 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.860766 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9908a34-d679-49a6-9c97-0a2ebf736e1b-workload-socket\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:10:41.861134 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.860785 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9908a34-d679-49a6-9c97-0a2ebf736e1b-credential-socket\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:10:41.861134 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.860799 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9908a34-d679-49a6-9c97-0a2ebf736e1b-workload-certs\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:10:41.861134 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.860813 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9908a34-d679-49a6-9c97-0a2ebf736e1b-istio-data\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:10:41.861134 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.860833 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9908a34-d679-49a6-9c97-0a2ebf736e1b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b9908a34-d679-49a6-9c97-0a2ebf736e1b" (UID: "b9908a34-d679-49a6-9c97-0a2ebf736e1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:10:41.862468 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.862443 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9908a34-d679-49a6-9c97-0a2ebf736e1b-kube-api-access-mhg2x" (OuterVolumeSpecName: "kube-api-access-mhg2x") pod "b9908a34-d679-49a6-9c97-0a2ebf736e1b" (UID: "b9908a34-d679-49a6-9c97-0a2ebf736e1b"). InnerVolumeSpecName "kube-api-access-mhg2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:10:41.862468 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.862451 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9908a34-d679-49a6-9c97-0a2ebf736e1b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b9908a34-d679-49a6-9c97-0a2ebf736e1b" (UID: "b9908a34-d679-49a6-9c97-0a2ebf736e1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 11:10:41.862929 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.862910 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9908a34-d679-49a6-9c97-0a2ebf736e1b-istio-token" (OuterVolumeSpecName: "istio-token") pod "b9908a34-d679-49a6-9c97-0a2ebf736e1b" (UID: "b9908a34-d679-49a6-9c97-0a2ebf736e1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:10:41.862998 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.862909 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9908a34-d679-49a6-9c97-0a2ebf736e1b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b9908a34-d679-49a6-9c97-0a2ebf736e1b" (UID: "b9908a34-d679-49a6-9c97-0a2ebf736e1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:10:41.926301 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.926211 2580 generic.go:356] "Generic (PLEG): container finished" podID="b9908a34-d679-49a6-9c97-0a2ebf736e1b" containerID="898668c7d4e1ce22dc5daaa1681925013b662defb2c8f371541d6fed6a62d3a0" exitCode=0 Aug 04 11:10:41.926301 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.926262 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" Aug 04 11:10:41.926482 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.926295 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" event={"ID":"b9908a34-d679-49a6-9c97-0a2ebf736e1b","Type":"ContainerDied","Data":"898668c7d4e1ce22dc5daaa1681925013b662defb2c8f371541d6fed6a62d3a0"} Aug 04 11:10:41.926482 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.926375 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg" event={"ID":"b9908a34-d679-49a6-9c97-0a2ebf736e1b","Type":"ContainerDied","Data":"d002f68e4073e99946d79cf781e2db697d44a3581e534ded65906b5ff8716f3e"} Aug 04 11:10:41.926482 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.926404 2580 scope.go:117] "RemoveContainer" containerID="898668c7d4e1ce22dc5daaa1681925013b662defb2c8f371541d6fed6a62d3a0" Aug 04 11:10:41.936157 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.936138 2580 scope.go:117] "RemoveContainer" containerID="c2e20ecc8da91eef7bf15a4ca3ffa3cf11e5fb983e563295f976ff09b87f2088" Aug 04 11:10:41.944399 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.944381 2580 scope.go:117] "RemoveContainer" containerID="898668c7d4e1ce22dc5daaa1681925013b662defb2c8f371541d6fed6a62d3a0" Aug 04 11:10:41.944715 ip-10-0-142-113 kubenswrapper[2580]: E0804 11:10:41.944696 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"898668c7d4e1ce22dc5daaa1681925013b662defb2c8f371541d6fed6a62d3a0\": container with ID starting with 898668c7d4e1ce22dc5daaa1681925013b662defb2c8f371541d6fed6a62d3a0 not found: ID does not exist" containerID="898668c7d4e1ce22dc5daaa1681925013b662defb2c8f371541d6fed6a62d3a0" Aug 04 11:10:41.944814 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.944727 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"898668c7d4e1ce22dc5daaa1681925013b662defb2c8f371541d6fed6a62d3a0"} err="failed to get container status \"898668c7d4e1ce22dc5daaa1681925013b662defb2c8f371541d6fed6a62d3a0\": rpc error: code = NotFound desc = could not find container \"898668c7d4e1ce22dc5daaa1681925013b662defb2c8f371541d6fed6a62d3a0\": container with ID starting with 898668c7d4e1ce22dc5daaa1681925013b662defb2c8f371541d6fed6a62d3a0 not found: ID does not exist" Aug 04 11:10:41.944814 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.944745 2580 scope.go:117] "RemoveContainer" containerID="c2e20ecc8da91eef7bf15a4ca3ffa3cf11e5fb983e563295f976ff09b87f2088" Aug 04 11:10:41.945023 ip-10-0-142-113 kubenswrapper[2580]: E0804 11:10:41.945004 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2e20ecc8da91eef7bf15a4ca3ffa3cf11e5fb983e563295f976ff09b87f2088\": container with ID starting with c2e20ecc8da91eef7bf15a4ca3ffa3cf11e5fb983e563295f976ff09b87f2088 not found: ID does not exist" containerID="c2e20ecc8da91eef7bf15a4ca3ffa3cf11e5fb983e563295f976ff09b87f2088" Aug 04 11:10:41.945067 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.945028 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2e20ecc8da91eef7bf15a4ca3ffa3cf11e5fb983e563295f976ff09b87f2088"} err="failed to get container status \"c2e20ecc8da91eef7bf15a4ca3ffa3cf11e5fb983e563295f976ff09b87f2088\": rpc error: code = NotFound desc = could not find container \"c2e20ecc8da91eef7bf15a4ca3ffa3cf11e5fb983e563295f976ff09b87f2088\": container with ID starting with c2e20ecc8da91eef7bf15a4ca3ffa3cf11e5fb983e563295f976ff09b87f2088 not found: ID does not exist" Aug 04 11:10:41.949743 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.949719 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg"] Aug 04 11:10:41.952636 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.952602 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-1c7c26-openshift-default-7d679b7cb4-njrwg"] Aug 04 11:10:41.961574 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.961539 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9908a34-d679-49a6-9c97-0a2ebf736e1b-istio-envoy\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:10:41.961696 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.961602 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9908a34-d679-49a6-9c97-0a2ebf736e1b-istio-token\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:10:41.961696 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.961618 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9908a34-d679-49a6-9c97-0a2ebf736e1b-istio-podinfo\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:10:41.961696 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.961632 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhg2x\" (UniqueName: \"kubernetes.io/projected/b9908a34-d679-49a6-9c97-0a2ebf736e1b-kube-api-access-mhg2x\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:10:41.961696 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:41.961646 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9908a34-d679-49a6-9c97-0a2ebf736e1b-istiod-ca-cert\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:10:43.100253 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:43.100208 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9908a34-d679-49a6-9c97-0a2ebf736e1b" path="/var/lib/kubelet/pods/b9908a34-d679-49a6-9c97-0a2ebf736e1b/volumes" Aug 04 11:10:44.435534 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:44.435497 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:10:54.917352 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:54.917315 2580 patch_prober.go:28] interesting pod/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Aug 04 11:10:54.917836 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:54.917376 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" podUID="81cf8e38-9dff-40c5-936c-d48485868cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Aug 04 11:10:54.988484 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:54.988450 2580 generic.go:356] "Generic (PLEG): container finished" podID="81cf8e38-9dff-40c5-936c-d48485868cde" containerID="89697c93f83c8125ca9addb09c6e6bf37a6c47693e551662dba022309aabd332" exitCode=0 Aug 04 11:10:54.988681 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:54.988519 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" event={"ID":"81cf8e38-9dff-40c5-936c-d48485868cde","Type":"ContainerDied","Data":"89697c93f83c8125ca9addb09c6e6bf37a6c47693e551662dba022309aabd332"} Aug 04 11:10:54.988947 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:54.988931 2580 scope.go:117] "RemoveContainer" containerID="89697c93f83c8125ca9addb09c6e6bf37a6c47693e551662dba022309aabd332" Aug 04 11:10:54.989935 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:54.989900 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:10:54.990044 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:54.989982 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:10:54.990044 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:54.990034 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:10:55.998785 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:55.998740 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" event={"ID":"81cf8e38-9dff-40c5-936c-d48485868cde","Type":"ContainerStarted","Data":"457f5b21c856e021195f402966709141bc161af62b81bbc847715b21b9d8692b"} Aug 04 11:10:56.718117 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:56.718078 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:10:56.718304 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:56.718288 2580 patch_prober.go:28] interesting pod/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Aug 04 11:10:56.718355 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:56.718336 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" podUID="81cf8e38-9dff-40c5-936c-d48485868cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Aug 04 11:10:57.718965 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:57.718925 2580 patch_prober.go:28] interesting pod/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Aug 04 11:10:57.719371 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:57.718986 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" podUID="81cf8e38-9dff-40c5-936c-d48485868cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Aug 04 11:10:58.718454 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:58.718410 2580 patch_prober.go:28] interesting pod/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Aug 04 11:10:58.718679 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:58.718477 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" podUID="81cf8e38-9dff-40c5-936c-d48485868cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Aug 04 11:10:59.718144 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:59.718107 2580 patch_prober.go:28] interesting pod/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Aug 04 11:10:59.718543 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:10:59.718163 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" podUID="81cf8e38-9dff-40c5-936c-d48485868cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Aug 04 11:11:00.718079 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:00.718044 2580 patch_prober.go:28] interesting pod/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Aug 04 11:11:00.718251 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:00.718100 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" podUID="81cf8e38-9dff-40c5-936c-d48485868cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Aug 04 11:11:01.722683 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:01.722604 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:11:01.723064 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:01.722982 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:11:01.723701 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:01.723684 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:11:06.662922 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.662887 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl"] Aug 04 11:11:06.663475 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.663376 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9908a34-d679-49a6-9c97-0a2ebf736e1b" containerName="istio-proxy" Aug 04 11:11:06.663475 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.663397 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9908a34-d679-49a6-9c97-0a2ebf736e1b" containerName="istio-proxy" Aug 04 11:11:06.663475 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.663416 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9908a34-d679-49a6-9c97-0a2ebf736e1b" containerName="istio-proxy" Aug 04 11:11:06.663475 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.663425 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9908a34-d679-49a6-9c97-0a2ebf736e1b" containerName="istio-proxy" Aug 04 11:11:06.663738 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.663522 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9908a34-d679-49a6-9c97-0a2ebf736e1b" containerName="istio-proxy" Aug 04 11:11:06.663738 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.663536 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9908a34-d679-49a6-9c97-0a2ebf736e1b" containerName="istio-proxy" Aug 04 11:11:06.666811 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.666787 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:06.669521 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.669499 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-74c752-openshift-default-dockercfg-mldcx\"" Aug 04 11:11:06.677331 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.677308 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl"] Aug 04 11:11:06.782770 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.782738 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ddf8176-6cda-4603-a71f-3d7ba8361a98-workload-certs\") pod \"e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:06.782974 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.782787 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ddf8176-6cda-4603-a71f-3d7ba8361a98-istio-envoy\") pod \"e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:06.782974 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.782903 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddf8176-6cda-4603-a71f-3d7ba8361a98-credential-socket\") pod \"e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:06.782974 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.782964 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ddf8176-6cda-4603-a71f-3d7ba8361a98-istio-podinfo\") pod \"e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:06.783163 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.783000 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ddf8176-6cda-4603-a71f-3d7ba8361a98-istio-data\") pod \"e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:06.783163 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.783041 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ddf8176-6cda-4603-a71f-3d7ba8361a98-istio-token\") pod \"e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:06.783163 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.783094 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddf8176-6cda-4603-a71f-3d7ba8361a98-workload-socket\") pod \"e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:06.783163 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.783122 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmb4w\" (UniqueName: \"kubernetes.io/projected/5ddf8176-6cda-4603-a71f-3d7ba8361a98-kube-api-access-lmb4w\") pod \"e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:06.783322 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.783163 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ddf8176-6cda-4603-a71f-3d7ba8361a98-istiod-ca-cert\") pod \"e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:06.884609 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.884574 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddf8176-6cda-4603-a71f-3d7ba8361a98-workload-socket\") pod \"e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:06.884609 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.884608 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmb4w\" (UniqueName: \"kubernetes.io/projected/5ddf8176-6cda-4603-a71f-3d7ba8361a98-kube-api-access-lmb4w\") pod \"e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:06.884850 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.884640 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ddf8176-6cda-4603-a71f-3d7ba8361a98-istiod-ca-cert\") pod \"e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:06.884850 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.884677 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ddf8176-6cda-4603-a71f-3d7ba8361a98-workload-certs\") pod \"e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:06.884850 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.884701 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ddf8176-6cda-4603-a71f-3d7ba8361a98-istio-envoy\") pod \"e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:06.884850 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.884730 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddf8176-6cda-4603-a71f-3d7ba8361a98-credential-socket\") pod \"e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:06.884850 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.884750 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ddf8176-6cda-4603-a71f-3d7ba8361a98-istio-podinfo\") pod \"e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:06.884850 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.884782 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ddf8176-6cda-4603-a71f-3d7ba8361a98-istio-data\") pod \"e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:06.884850 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.884812 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ddf8176-6cda-4603-a71f-3d7ba8361a98-istio-token\") pod \"e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:06.885188 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.885119 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ddf8176-6cda-4603-a71f-3d7ba8361a98-workload-certs\") pod \"e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:06.885188 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.885177 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddf8176-6cda-4603-a71f-3d7ba8361a98-credential-socket\") pod \"e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:06.885278 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.885244 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ddf8176-6cda-4603-a71f-3d7ba8361a98-istio-data\") pod \"e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:06.885409 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.885388 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddf8176-6cda-4603-a71f-3d7ba8361a98-workload-socket\") pod \"e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:06.885468 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.885413 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ddf8176-6cda-4603-a71f-3d7ba8361a98-istiod-ca-cert\") pod \"e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:06.887302 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.887266 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ddf8176-6cda-4603-a71f-3d7ba8361a98-istio-envoy\") pod \"e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:06.887482 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.887463 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ddf8176-6cda-4603-a71f-3d7ba8361a98-istio-podinfo\") pod \"e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:06.893316 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.893288 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ddf8176-6cda-4603-a71f-3d7ba8361a98-istio-token\") pod \"e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:06.893408 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.893370 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmb4w\" (UniqueName: \"kubernetes.io/projected/5ddf8176-6cda-4603-a71f-3d7ba8361a98-kube-api-access-lmb4w\") pod \"e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:06.978139 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:06.978022 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:07.128197 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:07.128169 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl"] Aug 04 11:11:07.130373 ip-10-0-142-113 kubenswrapper[2580]: W0804 11:11:07.130344 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ddf8176_6cda_4603_a71f_3d7ba8361a98.slice/crio-0fb9ed74c0b2858f9e0465d48a8b350e09104f1807f357748c718f00555446ad WatchSource:0}: Error finding container 0fb9ed74c0b2858f9e0465d48a8b350e09104f1807f357748c718f00555446ad: Status 404 returned error can't find the container with id 0fb9ed74c0b2858f9e0465d48a8b350e09104f1807f357748c718f00555446ad Aug 04 11:11:07.132898 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:07.132860 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:11:07.133011 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:07.132940 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:11:07.133011 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:07.132984 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:11:08.057011 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:08.056972 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" event={"ID":"5ddf8176-6cda-4603-a71f-3d7ba8361a98","Type":"ContainerStarted","Data":"0d325bffc321ebaab0ad85f74e59ca874a27e95297721633989005f76aa08968"} Aug 04 11:11:08.057492 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:08.057023 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" event={"ID":"5ddf8176-6cda-4603-a71f-3d7ba8361a98","Type":"ContainerStarted","Data":"0fb9ed74c0b2858f9e0465d48a8b350e09104f1807f357748c718f00555446ad"} Aug 04 11:11:08.081174 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:08.081105 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" podStartSLOduration=2.081084666 podStartE2EDuration="2.081084666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:11:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:11:08.078530545 +0000 UTC m=+2607.499417957" watchObservedRunningTime="2026-08-04 11:11:08.081084666 +0000 UTC m=+2607.501972078" Aug 04 11:11:08.978419 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:08.978388 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:08.983085 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:08.983058 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:09.061976 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:09.061937 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:09.063006 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:09.062989 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:12.533165 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:12.533123 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:11:22.138517 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:22.138485 2580 generic.go:356] "Generic (PLEG): container finished" podID="5ddf8176-6cda-4603-a71f-3d7ba8361a98" containerID="0d325bffc321ebaab0ad85f74e59ca874a27e95297721633989005f76aa08968" exitCode=0 Aug 04 11:11:22.138950 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:22.138553 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" event={"ID":"5ddf8176-6cda-4603-a71f-3d7ba8361a98","Type":"ContainerDied","Data":"0d325bffc321ebaab0ad85f74e59ca874a27e95297721633989005f76aa08968"} Aug 04 11:11:22.138998 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:22.138963 2580 scope.go:117] "RemoveContainer" containerID="0d325bffc321ebaab0ad85f74e59ca874a27e95297721633989005f76aa08968" Aug 04 11:11:22.139970 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:22.139941 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:11:22.140065 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:22.140005 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:11:22.140065 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:22.140036 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:11:23.144833 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:23.144800 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" event={"ID":"5ddf8176-6cda-4603-a71f-3d7ba8361a98","Type":"ContainerStarted","Data":"3ac6a1699edadfb902d8b99e88b9894d041a8331088a85eb8e453f5d2b0fb73f"} Aug 04 11:11:23.980278 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:23.980230 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:23.980580 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:23.980524 2580 patch_prober.go:28] interesting pod/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Aug 04 11:11:23.980682 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:23.980623 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" podUID="5ddf8176-6cda-4603-a71f-3d7ba8361a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Aug 04 11:11:24.062752 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:24.062711 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:24.978803 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:24.978757 2580 patch_prober.go:28] interesting pod/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Aug 04 11:11:24.979303 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:24.978833 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" podUID="5ddf8176-6cda-4603-a71f-3d7ba8361a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Aug 04 11:11:25.978706 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:25.978663 2580 patch_prober.go:28] interesting pod/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Aug 04 11:11:25.978919 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:25.978734 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" podUID="5ddf8176-6cda-4603-a71f-3d7ba8361a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Aug 04 11:11:26.979234 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:26.979191 2580 patch_prober.go:28] interesting pod/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Aug 04 11:11:26.979750 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:26.979263 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" podUID="5ddf8176-6cda-4603-a71f-3d7ba8361a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Aug 04 11:11:27.978735 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:27.978696 2580 patch_prober.go:28] interesting pod/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Aug 04 11:11:27.979020 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:27.978991 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" podUID="5ddf8176-6cda-4603-a71f-3d7ba8361a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Aug 04 11:11:28.982075 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:28.982039 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:28.983021 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:28.983001 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:11:43.243173 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:43.243142 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1154a6-7wzt6"] Aug 04 11:11:43.247696 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:43.247679 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1154a6-7wzt6" Aug 04 11:11:43.250594 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:43.250575 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 11:11:43.250704 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:43.250687 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qptbt\"" Aug 04 11:11:43.255964 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:43.255935 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1154a6-7wzt6"] Aug 04 11:11:43.326362 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:43.326317 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e1c893-f4ed-4bda-b471-ff51b78c5283-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1154a6-7wzt6\" (UID: \"16e1c893-f4ed-4bda-b471-ff51b78c5283\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1154a6-7wzt6" Aug 04 11:11:43.427883 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:43.427843 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e1c893-f4ed-4bda-b471-ff51b78c5283-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1154a6-7wzt6\" (UID: \"16e1c893-f4ed-4bda-b471-ff51b78c5283\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1154a6-7wzt6" Aug 04 11:11:43.428503 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:43.428481 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e1c893-f4ed-4bda-b471-ff51b78c5283-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1154a6-7wzt6\" (UID: \"16e1c893-f4ed-4bda-b471-ff51b78c5283\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1154a6-7wzt6" Aug 04 11:11:43.558995 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:43.558887 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1154a6-7wzt6" Aug 04 11:11:43.696841 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:43.696814 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1154a6-7wzt6"] Aug 04 11:11:43.699348 ip-10-0-142-113 kubenswrapper[2580]: W0804 11:11:43.699312 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e1c893_f4ed_4bda_b471_ff51b78c5283.slice/crio-a96841beb2d0da19970ca0c20847e841e17da6bb6eea84efa77f990cb29154a6 WatchSource:0}: Error finding container a96841beb2d0da19970ca0c20847e841e17da6bb6eea84efa77f990cb29154a6: Status 404 returned error can't find the container with id a96841beb2d0da19970ca0c20847e841e17da6bb6eea84efa77f990cb29154a6 Aug 04 11:11:43.701757 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:43.701738 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 11:11:44.240136 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:44.240102 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1154a6-7wzt6" event={"ID":"16e1c893-f4ed-4bda-b471-ff51b78c5283","Type":"ContainerStarted","Data":"a96841beb2d0da19970ca0c20847e841e17da6bb6eea84efa77f990cb29154a6"} Aug 04 11:11:46.251629 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:46.251596 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1154a6-7wzt6" event={"ID":"16e1c893-f4ed-4bda-b471-ff51b78c5283","Type":"ContainerStarted","Data":"753116c0ecbcdae9656ebb8d16f68bd3d2548feddbecd71c2869c1c195ac67f5"} Aug 04 11:11:46.271506 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:46.271459 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1154a6-7wzt6" podStartSLOduration=1.022559551 podStartE2EDuration="3.271443047s" totalImagesPullingTime="2.248883496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:11:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:11:46.270097944 +0000 UTC m=+2645.690985357" watchObservedRunningTime="2026-08-04 11:11:46.271443047 +0000 UTC m=+2645.692330458" Aug 04 11:11:47.257797 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:47.257765 2580 generic.go:356] "Generic (PLEG): container finished" podID="16e1c893-f4ed-4bda-b471-ff51b78c5283" containerID="753116c0ecbcdae9656ebb8d16f68bd3d2548feddbecd71c2869c1c195ac67f5" exitCode=0 Aug 04 11:11:47.258293 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:47.257816 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1154a6-7wzt6" event={"ID":"16e1c893-f4ed-4bda-b471-ff51b78c5283","Type":"ContainerDied","Data":"753116c0ecbcdae9656ebb8d16f68bd3d2548feddbecd71c2869c1c195ac67f5"} Aug 04 11:11:48.399904 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:48.399876 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1154a6-7wzt6" Aug 04 11:11:48.472012 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:48.471966 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e1c893-f4ed-4bda-b471-ff51b78c5283-maas-api-service-ca\") pod \"16e1c893-f4ed-4bda-b471-ff51b78c5283\" (UID: \"16e1c893-f4ed-4bda-b471-ff51b78c5283\") " Aug 04 11:11:48.472341 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:48.472317 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e1c893-f4ed-4bda-b471-ff51b78c5283-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "16e1c893-f4ed-4bda-b471-ff51b78c5283" (UID: "16e1c893-f4ed-4bda-b471-ff51b78c5283"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:11:48.573340 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:48.573242 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e1c893-f4ed-4bda-b471-ff51b78c5283-maas-api-service-ca\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:11:49.268004 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:49.267975 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1154a6-7wzt6" Aug 04 11:11:49.268210 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:49.268012 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1154a6-7wzt6" event={"ID":"16e1c893-f4ed-4bda-b471-ff51b78c5283","Type":"ContainerDied","Data":"a96841beb2d0da19970ca0c20847e841e17da6bb6eea84efa77f990cb29154a6"} Aug 04 11:11:49.268210 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:49.268054 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a96841beb2d0da19970ca0c20847e841e17da6bb6eea84efa77f990cb29154a6" Aug 04 11:11:58.881914 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:58.881866 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1154a6-7wzt6"] Aug 04 11:11:58.884373 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:58.884348 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1154a6-7wzt6"] Aug 04 11:11:59.100749 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:11:59.100713 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16e1c893-f4ed-4bda-b471-ff51b78c5283" path="/var/lib/kubelet/pods/16e1c893-f4ed-4bda-b471-ff51b78c5283/volumes" Aug 04 11:12:01.230677 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:01.230638 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:12:05.604202 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:05.604164 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh"] Aug 04 11:12:05.604626 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:05.604435 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" podUID="81cf8e38-9dff-40c5-936c-d48485868cde" containerName="istio-proxy" containerID="cri-o://457f5b21c856e021195f402966709141bc161af62b81bbc847715b21b9d8692b" gracePeriod=30 Aug 04 11:12:10.873404 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:10.873343 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:12:10.974441 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:10.974405 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81cf8e38-9dff-40c5-936c-d48485868cde-credential-socket\") pod \"81cf8e38-9dff-40c5-936c-d48485868cde\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " Aug 04 11:12:10.974679 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:10.974470 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqdxj\" (UniqueName: \"kubernetes.io/projected/81cf8e38-9dff-40c5-936c-d48485868cde-kube-api-access-bqdxj\") pod \"81cf8e38-9dff-40c5-936c-d48485868cde\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " Aug 04 11:12:10.974679 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:10.974527 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81cf8e38-9dff-40c5-936c-d48485868cde-istio-token\") pod \"81cf8e38-9dff-40c5-936c-d48485868cde\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " Aug 04 11:12:10.974679 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:10.974548 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81cf8e38-9dff-40c5-936c-d48485868cde-istio-podinfo\") pod \"81cf8e38-9dff-40c5-936c-d48485868cde\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " Aug 04 11:12:10.974679 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:10.974631 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81cf8e38-9dff-40c5-936c-d48485868cde-istio-data\") pod \"81cf8e38-9dff-40c5-936c-d48485868cde\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " Aug 04 11:12:10.974679 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:10.974670 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81cf8e38-9dff-40c5-936c-d48485868cde-istiod-ca-cert\") pod \"81cf8e38-9dff-40c5-936c-d48485868cde\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " Aug 04 11:12:10.974953 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:10.974721 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81cf8e38-9dff-40c5-936c-d48485868cde-workload-certs\") pod \"81cf8e38-9dff-40c5-936c-d48485868cde\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " Aug 04 11:12:10.974953 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:10.974729 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81cf8e38-9dff-40c5-936c-d48485868cde-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "81cf8e38-9dff-40c5-936c-d48485868cde" (UID: "81cf8e38-9dff-40c5-936c-d48485868cde"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:12:10.974953 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:10.974754 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81cf8e38-9dff-40c5-936c-d48485868cde-istio-envoy\") pod \"81cf8e38-9dff-40c5-936c-d48485868cde\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " Aug 04 11:12:10.974953 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:10.974788 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81cf8e38-9dff-40c5-936c-d48485868cde-workload-socket\") pod \"81cf8e38-9dff-40c5-936c-d48485868cde\" (UID: \"81cf8e38-9dff-40c5-936c-d48485868cde\") " Aug 04 11:12:10.974953 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:10.974919 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81cf8e38-9dff-40c5-936c-d48485868cde-istio-data" (OuterVolumeSpecName: "istio-data") pod "81cf8e38-9dff-40c5-936c-d48485868cde" (UID: "81cf8e38-9dff-40c5-936c-d48485868cde"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:12:10.975211 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:10.974949 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81cf8e38-9dff-40c5-936c-d48485868cde-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "81cf8e38-9dff-40c5-936c-d48485868cde" (UID: "81cf8e38-9dff-40c5-936c-d48485868cde"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:12:10.975211 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:10.975082 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81cf8e38-9dff-40c5-936c-d48485868cde-credential-socket\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:12:10.975211 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:10.975097 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81cf8e38-9dff-40c5-936c-d48485868cde-istio-data\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:12:10.975211 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:10.975107 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81cf8e38-9dff-40c5-936c-d48485868cde-workload-certs\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:12:10.975211 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:10.975082 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81cf8e38-9dff-40c5-936c-d48485868cde-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "81cf8e38-9dff-40c5-936c-d48485868cde" (UID: "81cf8e38-9dff-40c5-936c-d48485868cde"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:12:10.975658 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:10.975314 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81cf8e38-9dff-40c5-936c-d48485868cde-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "81cf8e38-9dff-40c5-936c-d48485868cde" (UID: "81cf8e38-9dff-40c5-936c-d48485868cde"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:12:10.976920 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:10.976894 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81cf8e38-9dff-40c5-936c-d48485868cde-istio-token" (OuterVolumeSpecName: "istio-token") pod "81cf8e38-9dff-40c5-936c-d48485868cde" (UID: "81cf8e38-9dff-40c5-936c-d48485868cde"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:12:10.977018 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:10.976979 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81cf8e38-9dff-40c5-936c-d48485868cde-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "81cf8e38-9dff-40c5-936c-d48485868cde" (UID: "81cf8e38-9dff-40c5-936c-d48485868cde"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 11:12:10.977402 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:10.977385 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81cf8e38-9dff-40c5-936c-d48485868cde-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "81cf8e38-9dff-40c5-936c-d48485868cde" (UID: "81cf8e38-9dff-40c5-936c-d48485868cde"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:12:10.977458 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:10.977399 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81cf8e38-9dff-40c5-936c-d48485868cde-kube-api-access-bqdxj" (OuterVolumeSpecName: "kube-api-access-bqdxj") pod "81cf8e38-9dff-40c5-936c-d48485868cde" (UID: "81cf8e38-9dff-40c5-936c-d48485868cde"). InnerVolumeSpecName "kube-api-access-bqdxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:12:11.075737 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:11.075704 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81cf8e38-9dff-40c5-936c-d48485868cde-istio-token\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:12:11.075737 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:11.075736 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81cf8e38-9dff-40c5-936c-d48485868cde-istio-podinfo\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:12:11.075940 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:11.075748 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81cf8e38-9dff-40c5-936c-d48485868cde-istiod-ca-cert\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:12:11.075940 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:11.075760 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81cf8e38-9dff-40c5-936c-d48485868cde-istio-envoy\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:12:11.075940 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:11.075771 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81cf8e38-9dff-40c5-936c-d48485868cde-workload-socket\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:12:11.075940 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:11.075783 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqdxj\" (UniqueName: \"kubernetes.io/projected/81cf8e38-9dff-40c5-936c-d48485868cde-kube-api-access-bqdxj\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:12:11.379967 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:11.379928 2580 generic.go:356] "Generic (PLEG): container finished" podID="81cf8e38-9dff-40c5-936c-d48485868cde" containerID="457f5b21c856e021195f402966709141bc161af62b81bbc847715b21b9d8692b" exitCode=0 Aug 04 11:12:11.380164 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:11.379982 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" Aug 04 11:12:11.380164 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:11.380013 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" event={"ID":"81cf8e38-9dff-40c5-936c-d48485868cde","Type":"ContainerDied","Data":"457f5b21c856e021195f402966709141bc161af62b81bbc847715b21b9d8692b"} Aug 04 11:12:11.380164 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:11.380071 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh" event={"ID":"81cf8e38-9dff-40c5-936c-d48485868cde","Type":"ContainerDied","Data":"b525208d155007046d77f5d951c5fdcbffaafca9c0a294fa99bd7dc177d59fa1"} Aug 04 11:12:11.380164 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:11.380095 2580 scope.go:117] "RemoveContainer" containerID="457f5b21c856e021195f402966709141bc161af62b81bbc847715b21b9d8692b" Aug 04 11:12:11.394976 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:11.389619 2580 scope.go:117] "RemoveContainer" containerID="89697c93f83c8125ca9addb09c6e6bf37a6c47693e551662dba022309aabd332" Aug 04 11:12:11.399392 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:11.399369 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh"] Aug 04 11:12:11.403822 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:11.403795 2580 scope.go:117] "RemoveContainer" containerID="457f5b21c856e021195f402966709141bc161af62b81bbc847715b21b9d8692b" Aug 04 11:12:11.404127 ip-10-0-142-113 kubenswrapper[2580]: E0804 11:12:11.404106 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"457f5b21c856e021195f402966709141bc161af62b81bbc847715b21b9d8692b\": container with ID starting with 457f5b21c856e021195f402966709141bc161af62b81bbc847715b21b9d8692b not found: ID does not exist" containerID="457f5b21c856e021195f402966709141bc161af62b81bbc847715b21b9d8692b" Aug 04 11:12:11.404181 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:11.404136 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"457f5b21c856e021195f402966709141bc161af62b81bbc847715b21b9d8692b"} err="failed to get container status \"457f5b21c856e021195f402966709141bc161af62b81bbc847715b21b9d8692b\": rpc error: code = NotFound desc = could not find container \"457f5b21c856e021195f402966709141bc161af62b81bbc847715b21b9d8692b\": container with ID starting with 457f5b21c856e021195f402966709141bc161af62b81bbc847715b21b9d8692b not found: ID does not exist" Aug 04 11:12:11.404181 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:11.404154 2580 scope.go:117] "RemoveContainer" containerID="89697c93f83c8125ca9addb09c6e6bf37a6c47693e551662dba022309aabd332" Aug 04 11:12:11.404386 ip-10-0-142-113 kubenswrapper[2580]: E0804 11:12:11.404372 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89697c93f83c8125ca9addb09c6e6bf37a6c47693e551662dba022309aabd332\": container with ID starting with 89697c93f83c8125ca9addb09c6e6bf37a6c47693e551662dba022309aabd332 not found: ID does not exist" containerID="89697c93f83c8125ca9addb09c6e6bf37a6c47693e551662dba022309aabd332" Aug 04 11:12:11.404426 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:11.404389 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89697c93f83c8125ca9addb09c6e6bf37a6c47693e551662dba022309aabd332"} err="failed to get container status \"89697c93f83c8125ca9addb09c6e6bf37a6c47693e551662dba022309aabd332\": rpc error: code = NotFound desc = could not find container \"89697c93f83c8125ca9addb09c6e6bf37a6c47693e551662dba022309aabd332\": container with ID starting with 89697c93f83c8125ca9addb09c6e6bf37a6c47693e551662dba022309aabd332 not found: ID does not exist" Aug 04 11:12:11.405163 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:11.405143 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1154a6-openshift-default-6547464856-xlmqh"] Aug 04 11:12:13.099899 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:13.099866 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81cf8e38-9dff-40c5-936c-d48485868cde" path="/var/lib/kubelet/pods/81cf8e38-9dff-40c5-936c-d48485868cde/volumes" Aug 04 11:12:23.650762 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:23.650722 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:12:28.594344 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:28.594303 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl"] Aug 04 11:12:28.594850 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:28.594549 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" podUID="5ddf8176-6cda-4603-a71f-3d7ba8361a98" containerName="istio-proxy" containerID="cri-o://3ac6a1699edadfb902d8b99e88b9894d041a8331088a85eb8e453f5d2b0fb73f" gracePeriod=30 Aug 04 11:12:28.984989 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:28.984951 2580 patch_prober.go:28] interesting pod/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 11:12:28.985173 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:28.985017 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" podUID="5ddf8176-6cda-4603-a71f-3d7ba8361a98" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 11:12:29.504100 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.504067 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg"] Aug 04 11:12:29.504424 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.504409 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81cf8e38-9dff-40c5-936c-d48485868cde" containerName="istio-proxy" Aug 04 11:12:29.504507 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.504427 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="81cf8e38-9dff-40c5-936c-d48485868cde" containerName="istio-proxy" Aug 04 11:12:29.504507 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.504434 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16e1c893-f4ed-4bda-b471-ff51b78c5283" containerName="revoke-keys" Aug 04 11:12:29.504507 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.504440 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="16e1c893-f4ed-4bda-b471-ff51b78c5283" containerName="revoke-keys" Aug 04 11:12:29.504507 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.504449 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81cf8e38-9dff-40c5-936c-d48485868cde" containerName="istio-proxy" Aug 04 11:12:29.504507 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.504455 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="81cf8e38-9dff-40c5-936c-d48485868cde" containerName="istio-proxy" Aug 04 11:12:29.504780 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.504516 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="81cf8e38-9dff-40c5-936c-d48485868cde" containerName="istio-proxy" Aug 04 11:12:29.504780 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.504526 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="16e1c893-f4ed-4bda-b471-ff51b78c5283" containerName="revoke-keys" Aug 04 11:12:29.511508 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.511486 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:29.514325 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.514306 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-bc4a78-openshift-default-dockercfg-wsg2l\"" Aug 04 11:12:29.524768 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.524683 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg"] Aug 04 11:12:29.532847 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.532822 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81863b6-bc8d-403a-9399-dff1f20d4c59-workload-socket\") pod \"e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:29.532968 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.532860 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81863b6-bc8d-403a-9399-dff1f20d4c59-workload-certs\") pod \"e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:29.532968 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.532887 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81863b6-bc8d-403a-9399-dff1f20d4c59-istio-envoy\") pod \"e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:29.532968 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.532905 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81863b6-bc8d-403a-9399-dff1f20d4c59-credential-socket\") pod \"e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:29.533100 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.532985 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81863b6-bc8d-403a-9399-dff1f20d4c59-istio-data\") pod \"e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:29.533100 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.533011 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24sdb\" (UniqueName: \"kubernetes.io/projected/d81863b6-bc8d-403a-9399-dff1f20d4c59-kube-api-access-24sdb\") pod \"e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:29.533100 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.533037 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81863b6-bc8d-403a-9399-dff1f20d4c59-istio-token\") pod \"e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:29.533100 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.533083 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81863b6-bc8d-403a-9399-dff1f20d4c59-istio-podinfo\") pod \"e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:29.533240 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.533109 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81863b6-bc8d-403a-9399-dff1f20d4c59-istiod-ca-cert\") pod \"e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:29.634132 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.634096 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81863b6-bc8d-403a-9399-dff1f20d4c59-istio-podinfo\") pod \"e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:29.634581 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.634146 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81863b6-bc8d-403a-9399-dff1f20d4c59-istiod-ca-cert\") pod \"e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:29.634581 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.634186 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81863b6-bc8d-403a-9399-dff1f20d4c59-workload-socket\") pod \"e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:29.634581 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.634217 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81863b6-bc8d-403a-9399-dff1f20d4c59-workload-certs\") pod \"e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:29.634581 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.634251 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81863b6-bc8d-403a-9399-dff1f20d4c59-istio-envoy\") pod \"e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:29.634581 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.634278 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81863b6-bc8d-403a-9399-dff1f20d4c59-credential-socket\") pod \"e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:29.634581 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.634312 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81863b6-bc8d-403a-9399-dff1f20d4c59-istio-data\") pod \"e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:29.634581 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.634335 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24sdb\" (UniqueName: \"kubernetes.io/projected/d81863b6-bc8d-403a-9399-dff1f20d4c59-kube-api-access-24sdb\") pod \"e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:29.634581 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.634365 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81863b6-bc8d-403a-9399-dff1f20d4c59-istio-token\") pod \"e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:29.635013 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.634803 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81863b6-bc8d-403a-9399-dff1f20d4c59-workload-socket\") pod \"e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:29.635013 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.634899 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81863b6-bc8d-403a-9399-dff1f20d4c59-credential-socket\") pod \"e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:29.635013 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.634988 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81863b6-bc8d-403a-9399-dff1f20d4c59-istio-data\") pod \"e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:29.635330 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.635036 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81863b6-bc8d-403a-9399-dff1f20d4c59-istiod-ca-cert\") pod \"e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:29.635330 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.635090 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81863b6-bc8d-403a-9399-dff1f20d4c59-workload-certs\") pod \"e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:29.636828 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.636804 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81863b6-bc8d-403a-9399-dff1f20d4c59-istio-podinfo\") pod \"e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:29.636941 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.636845 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81863b6-bc8d-403a-9399-dff1f20d4c59-istio-envoy\") pod \"e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:29.643399 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.643378 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81863b6-bc8d-403a-9399-dff1f20d4c59-istio-token\") pod \"e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:29.643729 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.643708 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24sdb\" (UniqueName: \"kubernetes.io/projected/d81863b6-bc8d-403a-9399-dff1f20d4c59-kube-api-access-24sdb\") pod \"e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:29.822249 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.822158 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:29.969537 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.969512 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg"] Aug 04 11:12:29.971419 ip-10-0-142-113 kubenswrapper[2580]: W0804 11:12:29.971395 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd81863b6_bc8d_403a_9399_dff1f20d4c59.slice/crio-daa4255ab5f413955ca9245989264c35884c86ec4cc0c7a7621f487cdc8007d7 WatchSource:0}: Error finding container daa4255ab5f413955ca9245989264c35884c86ec4cc0c7a7621f487cdc8007d7: Status 404 returned error can't find the container with id daa4255ab5f413955ca9245989264c35884c86ec4cc0c7a7621f487cdc8007d7 Aug 04 11:12:29.974113 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.974081 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:12:29.974195 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.974146 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:12:29.974195 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:29.974174 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:12:30.473227 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:30.473191 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" event={"ID":"d81863b6-bc8d-403a-9399-dff1f20d4c59","Type":"ContainerStarted","Data":"3ebae1001e6057b74a5a1ce893bd2bb50f9503954b3097e2bb0e899a7e24609d"} Aug 04 11:12:30.473400 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:30.473246 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" event={"ID":"d81863b6-bc8d-403a-9399-dff1f20d4c59","Type":"ContainerStarted","Data":"daa4255ab5f413955ca9245989264c35884c86ec4cc0c7a7621f487cdc8007d7"} Aug 04 11:12:30.496689 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:30.496575 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" podStartSLOduration=1.496538111 podStartE2EDuration="1.496538111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:12:30.496293354 +0000 UTC m=+2689.917180764" watchObservedRunningTime="2026-08-04 11:12:30.496538111 +0000 UTC m=+2689.917425524" Aug 04 11:12:30.823054 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:30.822962 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:31.828618 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:31.828533 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:32.485238 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:32.485202 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:32.486266 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:32.486245 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:33.847076 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:33.847050 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:12:33.874076 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:33.873999 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ddf8176-6cda-4603-a71f-3d7ba8361a98-istio-token\") pod \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " Aug 04 11:12:33.874076 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:33.874035 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmb4w\" (UniqueName: \"kubernetes.io/projected/5ddf8176-6cda-4603-a71f-3d7ba8361a98-kube-api-access-lmb4w\") pod \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " Aug 04 11:12:33.874295 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:33.874074 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ddf8176-6cda-4603-a71f-3d7ba8361a98-istio-envoy\") pod \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " Aug 04 11:12:33.874295 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:33.874105 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ddf8176-6cda-4603-a71f-3d7ba8361a98-istio-podinfo\") pod \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " Aug 04 11:12:33.874295 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:33.874124 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddf8176-6cda-4603-a71f-3d7ba8361a98-credential-socket\") pod \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " Aug 04 11:12:33.874295 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:33.874158 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ddf8176-6cda-4603-a71f-3d7ba8361a98-istio-data\") pod \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " Aug 04 11:12:33.874295 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:33.874182 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ddf8176-6cda-4603-a71f-3d7ba8361a98-workload-certs\") pod \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " Aug 04 11:12:33.874295 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:33.874235 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ddf8176-6cda-4603-a71f-3d7ba8361a98-istiod-ca-cert\") pod \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " Aug 04 11:12:33.874295 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:33.874297 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddf8176-6cda-4603-a71f-3d7ba8361a98-workload-socket\") pod \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\" (UID: \"5ddf8176-6cda-4603-a71f-3d7ba8361a98\") " Aug 04 11:12:33.875079 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:33.874747 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ddf8176-6cda-4603-a71f-3d7ba8361a98-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5ddf8176-6cda-4603-a71f-3d7ba8361a98" (UID: "5ddf8176-6cda-4603-a71f-3d7ba8361a98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:12:33.875079 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:33.875066 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ddf8176-6cda-4603-a71f-3d7ba8361a98-istio-data" (OuterVolumeSpecName: "istio-data") pod "5ddf8176-6cda-4603-a71f-3d7ba8361a98" (UID: "5ddf8176-6cda-4603-a71f-3d7ba8361a98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:12:33.875277 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:33.875081 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ddf8176-6cda-4603-a71f-3d7ba8361a98-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5ddf8176-6cda-4603-a71f-3d7ba8361a98" (UID: "5ddf8176-6cda-4603-a71f-3d7ba8361a98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:12:33.875277 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:33.875239 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ddf8176-6cda-4603-a71f-3d7ba8361a98-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5ddf8176-6cda-4603-a71f-3d7ba8361a98" (UID: "5ddf8176-6cda-4603-a71f-3d7ba8361a98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:12:33.875500 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:33.875466 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ddf8176-6cda-4603-a71f-3d7ba8361a98-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5ddf8176-6cda-4603-a71f-3d7ba8361a98" (UID: "5ddf8176-6cda-4603-a71f-3d7ba8361a98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:12:33.876790 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:33.876763 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ddf8176-6cda-4603-a71f-3d7ba8361a98-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5ddf8176-6cda-4603-a71f-3d7ba8361a98" (UID: "5ddf8176-6cda-4603-a71f-3d7ba8361a98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:12:33.876933 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:33.876904 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ddf8176-6cda-4603-a71f-3d7ba8361a98-kube-api-access-lmb4w" (OuterVolumeSpecName: "kube-api-access-lmb4w") pod "5ddf8176-6cda-4603-a71f-3d7ba8361a98" (UID: "5ddf8176-6cda-4603-a71f-3d7ba8361a98"). InnerVolumeSpecName "kube-api-access-lmb4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:12:33.877079 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:33.877060 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ddf8176-6cda-4603-a71f-3d7ba8361a98-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5ddf8176-6cda-4603-a71f-3d7ba8361a98" (UID: "5ddf8176-6cda-4603-a71f-3d7ba8361a98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 11:12:33.877147 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:33.877079 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ddf8176-6cda-4603-a71f-3d7ba8361a98-istio-token" (OuterVolumeSpecName: "istio-token") pod "5ddf8176-6cda-4603-a71f-3d7ba8361a98" (UID: "5ddf8176-6cda-4603-a71f-3d7ba8361a98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:12:33.975965 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:33.975924 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmb4w\" (UniqueName: \"kubernetes.io/projected/5ddf8176-6cda-4603-a71f-3d7ba8361a98-kube-api-access-lmb4w\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:12:33.975965 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:33.975961 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ddf8176-6cda-4603-a71f-3d7ba8361a98-istio-envoy\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:12:33.975965 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:33.975973 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ddf8176-6cda-4603-a71f-3d7ba8361a98-istio-podinfo\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:12:33.976221 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:33.975985 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddf8176-6cda-4603-a71f-3d7ba8361a98-credential-socket\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:12:33.976221 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:33.975996 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ddf8176-6cda-4603-a71f-3d7ba8361a98-istio-data\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:12:33.976221 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:33.976008 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ddf8176-6cda-4603-a71f-3d7ba8361a98-workload-certs\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:12:33.976221 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:33.976017 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ddf8176-6cda-4603-a71f-3d7ba8361a98-istiod-ca-cert\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:12:33.976221 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:33.976027 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddf8176-6cda-4603-a71f-3d7ba8361a98-workload-socket\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:12:33.976221 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:33.976040 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ddf8176-6cda-4603-a71f-3d7ba8361a98-istio-token\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:12:34.496678 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:34.496642 2580 generic.go:356] "Generic (PLEG): container finished" podID="5ddf8176-6cda-4603-a71f-3d7ba8361a98" containerID="3ac6a1699edadfb902d8b99e88b9894d041a8331088a85eb8e453f5d2b0fb73f" exitCode=0 Aug 04 11:12:34.496864 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:34.496691 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" Aug 04 11:12:34.496864 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:34.496722 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" event={"ID":"5ddf8176-6cda-4603-a71f-3d7ba8361a98","Type":"ContainerDied","Data":"3ac6a1699edadfb902d8b99e88b9894d041a8331088a85eb8e453f5d2b0fb73f"} Aug 04 11:12:34.496864 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:34.496780 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl" event={"ID":"5ddf8176-6cda-4603-a71f-3d7ba8361a98","Type":"ContainerDied","Data":"0fb9ed74c0b2858f9e0465d48a8b350e09104f1807f357748c718f00555446ad"} Aug 04 11:12:34.496864 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:34.496810 2580 scope.go:117] "RemoveContainer" containerID="3ac6a1699edadfb902d8b99e88b9894d041a8331088a85eb8e453f5d2b0fb73f" Aug 04 11:12:34.506888 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:34.506866 2580 scope.go:117] "RemoveContainer" containerID="0d325bffc321ebaab0ad85f74e59ca874a27e95297721633989005f76aa08968" Aug 04 11:12:34.515112 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:34.515092 2580 scope.go:117] "RemoveContainer" containerID="3ac6a1699edadfb902d8b99e88b9894d041a8331088a85eb8e453f5d2b0fb73f" Aug 04 11:12:34.515361 ip-10-0-142-113 kubenswrapper[2580]: E0804 11:12:34.515341 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ac6a1699edadfb902d8b99e88b9894d041a8331088a85eb8e453f5d2b0fb73f\": container with ID starting with 3ac6a1699edadfb902d8b99e88b9894d041a8331088a85eb8e453f5d2b0fb73f not found: ID does not exist" containerID="3ac6a1699edadfb902d8b99e88b9894d041a8331088a85eb8e453f5d2b0fb73f" Aug 04 11:12:34.515406 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:34.515371 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ac6a1699edadfb902d8b99e88b9894d041a8331088a85eb8e453f5d2b0fb73f"} err="failed to get container status \"3ac6a1699edadfb902d8b99e88b9894d041a8331088a85eb8e453f5d2b0fb73f\": rpc error: code = NotFound desc = could not find container \"3ac6a1699edadfb902d8b99e88b9894d041a8331088a85eb8e453f5d2b0fb73f\": container with ID starting with 3ac6a1699edadfb902d8b99e88b9894d041a8331088a85eb8e453f5d2b0fb73f not found: ID does not exist" Aug 04 11:12:34.515406 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:34.515392 2580 scope.go:117] "RemoveContainer" containerID="0d325bffc321ebaab0ad85f74e59ca874a27e95297721633989005f76aa08968" Aug 04 11:12:34.515626 ip-10-0-142-113 kubenswrapper[2580]: E0804 11:12:34.515600 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d325bffc321ebaab0ad85f74e59ca874a27e95297721633989005f76aa08968\": container with ID starting with 0d325bffc321ebaab0ad85f74e59ca874a27e95297721633989005f76aa08968 not found: ID does not exist" containerID="0d325bffc321ebaab0ad85f74e59ca874a27e95297721633989005f76aa08968" Aug 04 11:12:34.515706 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:34.515634 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d325bffc321ebaab0ad85f74e59ca874a27e95297721633989005f76aa08968"} err="failed to get container status \"0d325bffc321ebaab0ad85f74e59ca874a27e95297721633989005f76aa08968\": rpc error: code = NotFound desc = could not find container \"0d325bffc321ebaab0ad85f74e59ca874a27e95297721633989005f76aa08968\": container with ID starting with 0d325bffc321ebaab0ad85f74e59ca874a27e95297721633989005f76aa08968 not found: ID does not exist" Aug 04 11:12:34.519147 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:34.519126 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl"] Aug 04 11:12:34.522630 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:34.522608 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-74c752-openshift-default-6bf885965d-m5kzl"] Aug 04 11:12:35.099815 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:35.099783 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ddf8176-6cda-4603-a71f-3d7ba8361a98" path="/var/lib/kubelet/pods/5ddf8176-6cda-4603-a71f-3d7ba8361a98/volumes" Aug 04 11:12:36.226297 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:36.226265 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:12:41.227430 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:41.227395 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tb4zk_c5922147-cb42-4f62-9880-8deaa51a4b0f/ovn-acl-logging/0.log" Aug 04 11:12:41.230402 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:41.230376 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tb4zk_c5922147-cb42-4f62-9880-8deaa51a4b0f/ovn-acl-logging/0.log" Aug 04 11:12:45.555230 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:45.555129 2580 generic.go:356] "Generic (PLEG): container finished" podID="d81863b6-bc8d-403a-9399-dff1f20d4c59" containerID="3ebae1001e6057b74a5a1ce893bd2bb50f9503954b3097e2bb0e899a7e24609d" exitCode=0 Aug 04 11:12:45.555230 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:45.555205 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" event={"ID":"d81863b6-bc8d-403a-9399-dff1f20d4c59","Type":"ContainerDied","Data":"3ebae1001e6057b74a5a1ce893bd2bb50f9503954b3097e2bb0e899a7e24609d"} Aug 04 11:12:45.555702 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:45.555640 2580 scope.go:117] "RemoveContainer" containerID="3ebae1001e6057b74a5a1ce893bd2bb50f9503954b3097e2bb0e899a7e24609d" Aug 04 11:12:45.556670 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:45.556638 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:12:45.556753 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:45.556723 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:12:45.556796 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:45.556770 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:12:46.563072 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:46.563039 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" event={"ID":"d81863b6-bc8d-403a-9399-dff1f20d4c59","Type":"ContainerStarted","Data":"f33484a83fccb724ae79de0ab2742e4e940b2c8ed22952c6ff3e0db5e2150b24"} Aug 04 11:12:46.823311 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:46.823222 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:46.824484 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:46.824451 2580 patch_prober.go:28] interesting pod/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Aug 04 11:12:46.824652 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:46.824503 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" podUID="d81863b6-bc8d-403a-9399-dff1f20d4c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Aug 04 11:12:47.495724 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:47.495676 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:47.822683 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:47.822542 2580 patch_prober.go:28] interesting pod/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Aug 04 11:12:47.822683 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:47.822665 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" podUID="d81863b6-bc8d-403a-9399-dff1f20d4c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Aug 04 11:12:48.823083 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:48.823042 2580 patch_prober.go:28] interesting pod/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Aug 04 11:12:48.823488 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:48.823115 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" podUID="d81863b6-bc8d-403a-9399-dff1f20d4c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Aug 04 11:12:49.822627 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:49.822550 2580 patch_prober.go:28] interesting pod/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Aug 04 11:12:49.822836 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:49.822664 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" podUID="d81863b6-bc8d-403a-9399-dff1f20d4c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Aug 04 11:12:50.823458 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:50.823415 2580 patch_prober.go:28] interesting pod/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Aug 04 11:12:50.823895 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:50.823492 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" podUID="d81863b6-bc8d-403a-9399-dff1f20d4c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Aug 04 11:12:51.826391 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:51.826361 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:51.827288 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:51.827270 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:12:58.095892 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.095863 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w"] Aug 04 11:12:58.096257 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.096190 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ddf8176-6cda-4603-a71f-3d7ba8361a98" containerName="istio-proxy" Aug 04 11:12:58.096257 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.096201 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ddf8176-6cda-4603-a71f-3d7ba8361a98" containerName="istio-proxy" Aug 04 11:12:58.096334 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.096268 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="81cf8e38-9dff-40c5-936c-d48485868cde" containerName="istio-proxy" Aug 04 11:12:58.096334 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.096277 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ddf8176-6cda-4603-a71f-3d7ba8361a98" containerName="istio-proxy" Aug 04 11:12:58.096334 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.096284 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ddf8176-6cda-4603-a71f-3d7ba8361a98" containerName="istio-proxy" Aug 04 11:12:58.098468 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.098444 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:12:58.101258 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.101234 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-460cac-openshift-default-dockercfg-n9pwc\"" Aug 04 11:12:58.109968 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.109937 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w"] Aug 04 11:12:58.199351 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.199318 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4414c58-5988-4ed1-a563-403975e8e0d9-workload-certs\") pod \"e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:12:58.199351 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.199354 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq9z5\" (UniqueName: \"kubernetes.io/projected/a4414c58-5988-4ed1-a563-403975e8e0d9-kube-api-access-nq9z5\") pod \"e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:12:58.199596 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.199388 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4414c58-5988-4ed1-a563-403975e8e0d9-istio-envoy\") pod \"e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:12:58.199596 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.199404 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4414c58-5988-4ed1-a563-403975e8e0d9-istio-podinfo\") pod \"e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:12:58.199596 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.199422 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4414c58-5988-4ed1-a563-403975e8e0d9-istio-token\") pod \"e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:12:58.199596 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.199490 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4414c58-5988-4ed1-a563-403975e8e0d9-istiod-ca-cert\") pod \"e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:12:58.199596 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.199536 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4414c58-5988-4ed1-a563-403975e8e0d9-istio-data\") pod \"e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:12:58.199596 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.199593 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4414c58-5988-4ed1-a563-403975e8e0d9-workload-socket\") pod \"e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:12:58.199863 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.199618 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4414c58-5988-4ed1-a563-403975e8e0d9-credential-socket\") pod \"e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:12:58.300543 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.300506 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4414c58-5988-4ed1-a563-403975e8e0d9-istio-data\") pod \"e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:12:58.300752 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.300576 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4414c58-5988-4ed1-a563-403975e8e0d9-workload-socket\") pod \"e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:12:58.300752 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.300706 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4414c58-5988-4ed1-a563-403975e8e0d9-credential-socket\") pod \"e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:12:58.300879 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.300769 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4414c58-5988-4ed1-a563-403975e8e0d9-workload-certs\") pod \"e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:12:58.300879 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.300798 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq9z5\" (UniqueName: \"kubernetes.io/projected/a4414c58-5988-4ed1-a563-403975e8e0d9-kube-api-access-nq9z5\") pod \"e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:12:58.300879 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.300863 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4414c58-5988-4ed1-a563-403975e8e0d9-istio-envoy\") pod \"e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:12:58.301047 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.300893 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4414c58-5988-4ed1-a563-403975e8e0d9-istio-podinfo\") pod \"e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:12:58.301047 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.300917 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4414c58-5988-4ed1-a563-403975e8e0d9-istio-token\") pod \"e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:12:58.301047 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.300966 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4414c58-5988-4ed1-a563-403975e8e0d9-istio-data\") pod \"e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:12:58.301047 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.300970 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4414c58-5988-4ed1-a563-403975e8e0d9-istiod-ca-cert\") pod \"e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:12:58.301339 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.301311 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4414c58-5988-4ed1-a563-403975e8e0d9-credential-socket\") pod \"e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:12:58.301434 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.301315 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4414c58-5988-4ed1-a563-403975e8e0d9-workload-socket\") pod \"e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:12:58.301536 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.301516 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4414c58-5988-4ed1-a563-403975e8e0d9-workload-certs\") pod \"e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:12:58.302045 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.301979 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4414c58-5988-4ed1-a563-403975e8e0d9-istiod-ca-cert\") pod \"e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:12:58.304198 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.304173 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4414c58-5988-4ed1-a563-403975e8e0d9-istio-envoy\") pod \"e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:12:58.304325 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.304271 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4414c58-5988-4ed1-a563-403975e8e0d9-istio-podinfo\") pod \"e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:12:58.309287 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.309262 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4414c58-5988-4ed1-a563-403975e8e0d9-istio-token\") pod \"e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:12:58.309635 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.309612 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq9z5\" (UniqueName: \"kubernetes.io/projected/a4414c58-5988-4ed1-a563-403975e8e0d9-kube-api-access-nq9z5\") pod \"e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:12:58.408858 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.408826 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:12:58.551321 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.551295 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w"] Aug 04 11:12:58.553368 ip-10-0-142-113 kubenswrapper[2580]: W0804 11:12:58.553340 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4414c58_5988_4ed1_a563_403975e8e0d9.slice/crio-91687d50a92fe41b5138a51de11b341ebf51c345d3356bc0deb0736f3e4c2193 WatchSource:0}: Error finding container 91687d50a92fe41b5138a51de11b341ebf51c345d3356bc0deb0736f3e4c2193: Status 404 returned error can't find the container with id 91687d50a92fe41b5138a51de11b341ebf51c345d3356bc0deb0736f3e4c2193 Aug 04 11:12:58.555625 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.555595 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:12:58.555683 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.555666 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:12:58.555725 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.555697 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:12:58.627949 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.627913 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" event={"ID":"a4414c58-5988-4ed1-a563-403975e8e0d9","Type":"ContainerStarted","Data":"6799fd0b9e8dac093b6940309059a886ec0cf241e83de178b86fd1ab18ca19e8"} Aug 04 11:12:58.628123 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.627983 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" event={"ID":"a4414c58-5988-4ed1-a563-403975e8e0d9","Type":"ContainerStarted","Data":"91687d50a92fe41b5138a51de11b341ebf51c345d3356bc0deb0736f3e4c2193"} Aug 04 11:12:58.649767 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:58.649705 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" podStartSLOduration=0.649683099 podStartE2EDuration="649.683099ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:12:58.648901384 +0000 UTC m=+2718.069788797" watchObservedRunningTime="2026-08-04 11:12:58.649683099 +0000 UTC m=+2718.070570512" Aug 04 11:12:59.409342 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:59.409303 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:12:59.414087 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:59.414063 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:12:59.633480 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:59.633447 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:12:59.634452 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:12:59.634434 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:13:04.336155 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:04.336119 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:13:14.634766 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:14.634727 2580 patch_prober.go:28] interesting pod/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Aug 04 11:13:14.635196 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:14.634797 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" podUID="a4414c58-5988-4ed1-a563-403975e8e0d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Aug 04 11:13:14.711186 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:14.711151 2580 generic.go:356] "Generic (PLEG): container finished" podID="a4414c58-5988-4ed1-a563-403975e8e0d9" containerID="6799fd0b9e8dac093b6940309059a886ec0cf241e83de178b86fd1ab18ca19e8" exitCode=0 Aug 04 11:13:14.711364 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:14.711226 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" event={"ID":"a4414c58-5988-4ed1-a563-403975e8e0d9","Type":"ContainerDied","Data":"6799fd0b9e8dac093b6940309059a886ec0cf241e83de178b86fd1ab18ca19e8"} Aug 04 11:13:14.711669 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:14.711652 2580 scope.go:117] "RemoveContainer" containerID="6799fd0b9e8dac093b6940309059a886ec0cf241e83de178b86fd1ab18ca19e8" Aug 04 11:13:14.712742 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:14.712711 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:13:14.712837 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:14.712780 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:13:14.712837 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:14.712809 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:13:15.718125 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:15.718091 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" event={"ID":"a4414c58-5988-4ed1-a563-403975e8e0d9","Type":"ContainerStarted","Data":"2282034771a0f99f4e1b88a32b2ef1a548bddf2f8af710d5efb224c0cdf6b600"} Aug 04 11:13:16.409617 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:16.409541 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:13:16.409883 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:16.409737 2580 patch_prober.go:28] interesting pod/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Aug 04 11:13:16.409883 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:16.409777 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" podUID="a4414c58-5988-4ed1-a563-403975e8e0d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Aug 04 11:13:17.409919 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:17.409887 2580 patch_prober.go:28] interesting pod/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Aug 04 11:13:17.410297 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:17.409945 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" podUID="a4414c58-5988-4ed1-a563-403975e8e0d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Aug 04 11:13:18.409849 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:18.409806 2580 patch_prober.go:28] interesting pod/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Aug 04 11:13:18.410034 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:18.409870 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" podUID="a4414c58-5988-4ed1-a563-403975e8e0d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Aug 04 11:13:19.409412 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:19.409376 2580 patch_prober.go:28] interesting pod/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Aug 04 11:13:19.409615 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:19.409451 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" podUID="a4414c58-5988-4ed1-a563-403975e8e0d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Aug 04 11:13:20.409338 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:20.409301 2580 patch_prober.go:28] interesting pod/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Aug 04 11:13:20.409783 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:20.409366 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" podUID="a4414c58-5988-4ed1-a563-403975e8e0d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Aug 04 11:13:21.413863 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:21.413831 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:13:21.414308 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:21.414113 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:13:21.414919 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:21.414904 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:13:35.138883 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:35.138851 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q"] Aug 04 11:13:35.139261 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:35.139218 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ddf8176-6cda-4603-a71f-3d7ba8361a98" containerName="istio-proxy" Aug 04 11:13:35.139261 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:35.139233 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ddf8176-6cda-4603-a71f-3d7ba8361a98" containerName="istio-proxy" Aug 04 11:13:35.142306 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:35.142290 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q" Aug 04 11:13:35.145361 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:35.145339 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-bc4a78\"/\"openshift-service-ca.crt\"" Aug 04 11:13:35.145495 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:35.145375 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-bc4a78\"/\"kube-root-ca.crt\"" Aug 04 11:13:35.146772 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:35.146743 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-bc4a78\"/\"default-dockercfg-xmdqk\"" Aug 04 11:13:35.146876 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:35.146780 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-bc4a78\"/\"auth-test-model-bc4a78-kserve-self-signed-certs\"" Aug 04 11:13:35.152734 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:35.152707 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q"] Aug 04 11:13:35.239319 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:35.239276 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-tls-certs\") pod \"auth-test-model-bc4a78-kserve-5b6b4bf467-x524q\" (UID: \"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1\") " pod="ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q" Aug 04 11:13:35.239319 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:35.239327 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-dshm\") pod \"auth-test-model-bc4a78-kserve-5b6b4bf467-x524q\" (UID: \"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1\") " pod="ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q" Aug 04 11:13:35.239639 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:35.239362 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-home\") pod \"auth-test-model-bc4a78-kserve-5b6b4bf467-x524q\" (UID: \"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1\") " pod="ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q" Aug 04 11:13:35.239639 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:35.239385 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-model-cache\") pod \"auth-test-model-bc4a78-kserve-5b6b4bf467-x524q\" (UID: \"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1\") " pod="ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q" Aug 04 11:13:35.239639 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:35.239404 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-tmp-dir\") pod \"auth-test-model-bc4a78-kserve-5b6b4bf467-x524q\" (UID: \"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1\") " pod="ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q" Aug 04 11:13:35.239639 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:35.239512 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tqcg\" (UniqueName: \"kubernetes.io/projected/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-kube-api-access-5tqcg\") pod \"auth-test-model-bc4a78-kserve-5b6b4bf467-x524q\" (UID: \"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1\") " pod="ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q" Aug 04 11:13:35.340518 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:35.340469 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-tls-certs\") pod \"auth-test-model-bc4a78-kserve-5b6b4bf467-x524q\" (UID: \"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1\") " pod="ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q" Aug 04 11:13:35.340518 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:35.340529 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-dshm\") pod \"auth-test-model-bc4a78-kserve-5b6b4bf467-x524q\" (UID: \"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1\") " pod="ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q" Aug 04 11:13:35.340812 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:35.340596 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-home\") pod \"auth-test-model-bc4a78-kserve-5b6b4bf467-x524q\" (UID: \"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1\") " pod="ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q" Aug 04 11:13:35.340812 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:35.340620 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-model-cache\") pod \"auth-test-model-bc4a78-kserve-5b6b4bf467-x524q\" (UID: \"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1\") " pod="ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q" Aug 04 11:13:35.340812 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:35.340639 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-tmp-dir\") pod \"auth-test-model-bc4a78-kserve-5b6b4bf467-x524q\" (UID: \"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1\") " pod="ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q" Aug 04 11:13:35.340812 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:35.340688 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tqcg\" (UniqueName: \"kubernetes.io/projected/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-kube-api-access-5tqcg\") pod \"auth-test-model-bc4a78-kserve-5b6b4bf467-x524q\" (UID: \"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1\") " pod="ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q" Aug 04 11:13:35.341060 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:35.341038 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-home\") pod \"auth-test-model-bc4a78-kserve-5b6b4bf467-x524q\" (UID: \"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1\") " pod="ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q" Aug 04 11:13:35.341126 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:35.341067 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-model-cache\") pod \"auth-test-model-bc4a78-kserve-5b6b4bf467-x524q\" (UID: \"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1\") " pod="ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q" Aug 04 11:13:35.341184 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:35.341141 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-tmp-dir\") pod \"auth-test-model-bc4a78-kserve-5b6b4bf467-x524q\" (UID: \"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1\") " pod="ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q" Aug 04 11:13:35.343071 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:35.343052 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-dshm\") pod \"auth-test-model-bc4a78-kserve-5b6b4bf467-x524q\" (UID: \"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1\") " pod="ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q" Aug 04 11:13:35.343247 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:35.343228 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-tls-certs\") pod \"auth-test-model-bc4a78-kserve-5b6b4bf467-x524q\" (UID: \"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1\") " pod="ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q" Aug 04 11:13:35.351162 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:35.351137 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tqcg\" (UniqueName: \"kubernetes.io/projected/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-kube-api-access-5tqcg\") pod \"auth-test-model-bc4a78-kserve-5b6b4bf467-x524q\" (UID: \"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1\") " pod="ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q" Aug 04 11:13:35.451967 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:35.451884 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q" Aug 04 11:13:35.593410 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:35.593383 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q"] Aug 04 11:13:35.595541 ip-10-0-142-113 kubenswrapper[2580]: W0804 11:13:35.595512 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc25cb0e9_c5aa_46b5_be12_4f081cbce3b1.slice/crio-08132b91aaac7565271068518d6c557c9943f58f8ba2d79490c1353cbb65df27 WatchSource:0}: Error finding container 08132b91aaac7565271068518d6c557c9943f58f8ba2d79490c1353cbb65df27: Status 404 returned error can't find the container with id 08132b91aaac7565271068518d6c557c9943f58f8ba2d79490c1353cbb65df27 Aug 04 11:13:35.812497 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:35.812405 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q" event={"ID":"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1","Type":"ContainerStarted","Data":"64e0bdc15588559cf15c88e1b40fd4e9fd2ba34142ed3c7a79fe0cf39f54e2ab"} Aug 04 11:13:35.812497 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:35.812447 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q" event={"ID":"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1","Type":"ContainerStarted","Data":"08132b91aaac7565271068518d6c557c9943f58f8ba2d79490c1353cbb65df27"} Aug 04 11:13:35.831337 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:35.831286 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q" podStartSLOduration=0.831270671 podStartE2EDuration="831.270671ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:13:35.831241013 +0000 UTC m=+2755.252128424" watchObservedRunningTime="2026-08-04 11:13:35.831270671 +0000 UTC m=+2755.252158081" Aug 04 11:13:37.638361 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:37.638323 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:13:45.452651 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:45.452611 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q" Aug 04 11:13:45.453056 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:45.452693 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q" Aug 04 11:13:45.465460 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:45.465424 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q" Aug 04 11:13:45.870530 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:45.870498 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q" Aug 04 11:13:53.129891 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:53.129854 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk"] Aug 04 11:13:53.133474 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:53.133454 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk" Aug 04 11:13:53.136612 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:53.136582 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-460cac\"/\"openshift-service-ca.crt\"" Aug 04 11:13:53.136740 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:53.136619 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-460cac\"/\"kube-root-ca.crt\"" Aug 04 11:13:53.138050 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:53.138028 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-460cac\"/\"default-dockercfg-c7lzs\"" Aug 04 11:13:53.138154 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:53.138027 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-460cac\"/\"auth-test-model-460cac-kserve-self-signed-certs\"" Aug 04 11:13:53.145202 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:53.145176 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk"] Aug 04 11:13:53.210966 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:53.210932 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da54998d-9c50-4743-bc29-271cc2cfadfa-model-cache\") pod \"auth-test-model-460cac-kserve-7745b798dc-9m6bk\" (UID: \"da54998d-9c50-4743-bc29-271cc2cfadfa\") " pod="ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk" Aug 04 11:13:53.211151 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:53.210985 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjw9r\" (UniqueName: \"kubernetes.io/projected/da54998d-9c50-4743-bc29-271cc2cfadfa-kube-api-access-xjw9r\") pod \"auth-test-model-460cac-kserve-7745b798dc-9m6bk\" (UID: \"da54998d-9c50-4743-bc29-271cc2cfadfa\") " pod="ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk" Aug 04 11:13:53.211151 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:53.211089 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da54998d-9c50-4743-bc29-271cc2cfadfa-tmp-dir\") pod \"auth-test-model-460cac-kserve-7745b798dc-9m6bk\" (UID: \"da54998d-9c50-4743-bc29-271cc2cfadfa\") " pod="ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk" Aug 04 11:13:53.211151 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:53.211133 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da54998d-9c50-4743-bc29-271cc2cfadfa-tls-certs\") pod \"auth-test-model-460cac-kserve-7745b798dc-9m6bk\" (UID: \"da54998d-9c50-4743-bc29-271cc2cfadfa\") " pod="ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk" Aug 04 11:13:53.211263 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:53.211155 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da54998d-9c50-4743-bc29-271cc2cfadfa-dshm\") pod \"auth-test-model-460cac-kserve-7745b798dc-9m6bk\" (UID: \"da54998d-9c50-4743-bc29-271cc2cfadfa\") " pod="ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk" Aug 04 11:13:53.211263 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:53.211194 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da54998d-9c50-4743-bc29-271cc2cfadfa-home\") pod \"auth-test-model-460cac-kserve-7745b798dc-9m6bk\" (UID: \"da54998d-9c50-4743-bc29-271cc2cfadfa\") " pod="ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk" Aug 04 11:13:53.312334 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:53.312294 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da54998d-9c50-4743-bc29-271cc2cfadfa-tls-certs\") pod \"auth-test-model-460cac-kserve-7745b798dc-9m6bk\" (UID: \"da54998d-9c50-4743-bc29-271cc2cfadfa\") " pod="ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk" Aug 04 11:13:53.312334 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:53.312334 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da54998d-9c50-4743-bc29-271cc2cfadfa-dshm\") pod \"auth-test-model-460cac-kserve-7745b798dc-9m6bk\" (UID: \"da54998d-9c50-4743-bc29-271cc2cfadfa\") " pod="ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk" Aug 04 11:13:53.312610 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:53.312357 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da54998d-9c50-4743-bc29-271cc2cfadfa-home\") pod \"auth-test-model-460cac-kserve-7745b798dc-9m6bk\" (UID: \"da54998d-9c50-4743-bc29-271cc2cfadfa\") " pod="ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk" Aug 04 11:13:53.312610 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:53.312485 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da54998d-9c50-4743-bc29-271cc2cfadfa-model-cache\") pod \"auth-test-model-460cac-kserve-7745b798dc-9m6bk\" (UID: \"da54998d-9c50-4743-bc29-271cc2cfadfa\") " pod="ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk" Aug 04 11:13:53.312610 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:53.312599 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjw9r\" (UniqueName: \"kubernetes.io/projected/da54998d-9c50-4743-bc29-271cc2cfadfa-kube-api-access-xjw9r\") pod \"auth-test-model-460cac-kserve-7745b798dc-9m6bk\" (UID: \"da54998d-9c50-4743-bc29-271cc2cfadfa\") " pod="ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk" Aug 04 11:13:53.312731 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:53.312683 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da54998d-9c50-4743-bc29-271cc2cfadfa-tmp-dir\") pod \"auth-test-model-460cac-kserve-7745b798dc-9m6bk\" (UID: \"da54998d-9c50-4743-bc29-271cc2cfadfa\") " pod="ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk" Aug 04 11:13:53.312836 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:53.312819 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da54998d-9c50-4743-bc29-271cc2cfadfa-home\") pod \"auth-test-model-460cac-kserve-7745b798dc-9m6bk\" (UID: \"da54998d-9c50-4743-bc29-271cc2cfadfa\") " pod="ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk" Aug 04 11:13:53.312931 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:53.312911 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da54998d-9c50-4743-bc29-271cc2cfadfa-model-cache\") pod \"auth-test-model-460cac-kserve-7745b798dc-9m6bk\" (UID: \"da54998d-9c50-4743-bc29-271cc2cfadfa\") " pod="ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk" Aug 04 11:13:53.312983 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:53.312958 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da54998d-9c50-4743-bc29-271cc2cfadfa-tmp-dir\") pod \"auth-test-model-460cac-kserve-7745b798dc-9m6bk\" (UID: \"da54998d-9c50-4743-bc29-271cc2cfadfa\") " pod="ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk" Aug 04 11:13:53.314910 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:53.314882 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da54998d-9c50-4743-bc29-271cc2cfadfa-dshm\") pod \"auth-test-model-460cac-kserve-7745b798dc-9m6bk\" (UID: \"da54998d-9c50-4743-bc29-271cc2cfadfa\") " pod="ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk" Aug 04 11:13:53.315284 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:53.315256 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da54998d-9c50-4743-bc29-271cc2cfadfa-tls-certs\") pod \"auth-test-model-460cac-kserve-7745b798dc-9m6bk\" (UID: \"da54998d-9c50-4743-bc29-271cc2cfadfa\") " pod="ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk" Aug 04 11:13:53.322045 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:53.322019 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjw9r\" (UniqueName: \"kubernetes.io/projected/da54998d-9c50-4743-bc29-271cc2cfadfa-kube-api-access-xjw9r\") pod \"auth-test-model-460cac-kserve-7745b798dc-9m6bk\" (UID: \"da54998d-9c50-4743-bc29-271cc2cfadfa\") " pod="ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk" Aug 04 11:13:53.443306 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:53.443212 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk" Aug 04 11:13:53.590779 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:53.590745 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk"] Aug 04 11:13:53.593749 ip-10-0-142-113 kubenswrapper[2580]: W0804 11:13:53.593718 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda54998d_9c50_4743_bc29_271cc2cfadfa.slice/crio-f485faac45f8d7e7e673070672b11d02d47000c1ae772e36ceb1562391635355 WatchSource:0}: Error finding container f485faac45f8d7e7e673070672b11d02d47000c1ae772e36ceb1562391635355: Status 404 returned error can't find the container with id f485faac45f8d7e7e673070672b11d02d47000c1ae772e36ceb1562391635355 Aug 04 11:13:53.897973 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:53.897938 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk" event={"ID":"da54998d-9c50-4743-bc29-271cc2cfadfa","Type":"ContainerStarted","Data":"9094a8504c8a4e5e923d43937a742783fa2d3d5e9cac73c48e05b8029c63aca9"} Aug 04 11:13:53.898174 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:53.897983 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk" event={"ID":"da54998d-9c50-4743-bc29-271cc2cfadfa","Type":"ContainerStarted","Data":"f485faac45f8d7e7e673070672b11d02d47000c1ae772e36ceb1562391635355"} Aug 04 11:13:53.917406 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:53.917346 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk" podStartSLOduration=0.91732806 podStartE2EDuration="917.32806ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:13:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:13:53.915197031 +0000 UTC m=+2773.336084456" watchObservedRunningTime="2026-08-04 11:13:53.91732806 +0000 UTC m=+2773.338215508" Aug 04 11:13:55.828691 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:13:55.828653 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:14:03.444009 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:14:03.443913 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk" Aug 04 11:14:03.444009 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:14:03.443965 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk" Aug 04 11:14:03.456895 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:14:03.456865 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk" Aug 04 11:14:03.957709 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:14:03.957679 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk" Aug 04 11:14:26.040967 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:14:26.040924 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:14:41.538701 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:14:41.538666 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:14:53.433796 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:14:53.433755 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:15:08.231159 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:15:08.231116 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:15:19.334817 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:15:19.334779 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:15:34.826749 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:15:34.826708 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:15:46.729031 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:15:46.728992 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:16:01.331590 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:16:01.331539 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:16:13.436888 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:16:13.436850 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:16:27.929348 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:16:27.929305 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:16:39.650926 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:16:39.650889 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:16:54.550871 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:16:54.550834 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:16:58.685970 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:16:58.685937 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q"] Aug 04 11:16:59.725606 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:16:59.725550 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk"] Aug 04 11:17:04.937242 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:04.937161 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:17:13.705762 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:13.705716 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q" podUID="c25cb0e9-c5aa-46b5-be12-4f081cbce3b1" containerName="main" containerID="cri-o://64e0bdc15588559cf15c88e1b40fd4e9fd2ba34142ed3c7a79fe0cf39f54e2ab" gracePeriod=45 Aug 04 11:17:13.869871 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:13.869836 2580 generic.go:356] "Generic (PLEG): container finished" podID="c25cb0e9-c5aa-46b5-be12-4f081cbce3b1" containerID="64e0bdc15588559cf15c88e1b40fd4e9fd2ba34142ed3c7a79fe0cf39f54e2ab" exitCode=0 Aug 04 11:17:13.870052 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:13.869917 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q" event={"ID":"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1","Type":"ContainerDied","Data":"64e0bdc15588559cf15c88e1b40fd4e9fd2ba34142ed3c7a79fe0cf39f54e2ab"} Aug 04 11:17:13.959302 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:13.959244 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q" Aug 04 11:17:14.043687 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:14.043646 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-tls-certs\") pod \"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1\" (UID: \"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1\") " Aug 04 11:17:14.043858 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:14.043745 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-model-cache\") pod \"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1\" (UID: \"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1\") " Aug 04 11:17:14.043858 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:14.043801 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-home\") pod \"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1\" (UID: \"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1\") " Aug 04 11:17:14.043858 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:14.043818 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tqcg\" (UniqueName: \"kubernetes.io/projected/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-kube-api-access-5tqcg\") pod \"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1\" (UID: \"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1\") " Aug 04 11:17:14.043858 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:14.043836 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-dshm\") pod \"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1\" (UID: \"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1\") " Aug 04 11:17:14.044071 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:14.043865 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-tmp-dir\") pod \"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1\" (UID: \"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1\") " Aug 04 11:17:14.044071 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:14.043948 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-model-cache" (OuterVolumeSpecName: "model-cache") pod "c25cb0e9-c5aa-46b5-be12-4f081cbce3b1" (UID: "c25cb0e9-c5aa-46b5-be12-4f081cbce3b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:17:14.044182 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:14.044166 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-home" (OuterVolumeSpecName: "home") pod "c25cb0e9-c5aa-46b5-be12-4f081cbce3b1" (UID: "c25cb0e9-c5aa-46b5-be12-4f081cbce3b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:17:14.044244 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:14.044176 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c25cb0e9-c5aa-46b5-be12-4f081cbce3b1" (UID: "c25cb0e9-c5aa-46b5-be12-4f081cbce3b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:17:14.044296 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:14.044243 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-model-cache\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:17:14.046060 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:14.046034 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c25cb0e9-c5aa-46b5-be12-4f081cbce3b1" (UID: "c25cb0e9-c5aa-46b5-be12-4f081cbce3b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:17:14.046231 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:14.046210 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-dshm" (OuterVolumeSpecName: "dshm") pod "c25cb0e9-c5aa-46b5-be12-4f081cbce3b1" (UID: "c25cb0e9-c5aa-46b5-be12-4f081cbce3b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:17:14.046282 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:14.046234 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-kube-api-access-5tqcg" (OuterVolumeSpecName: "kube-api-access-5tqcg") pod "c25cb0e9-c5aa-46b5-be12-4f081cbce3b1" (UID: "c25cb0e9-c5aa-46b5-be12-4f081cbce3b1"). InnerVolumeSpecName "kube-api-access-5tqcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:17:14.145136 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:14.145094 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-tmp-dir\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:17:14.145136 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:14.145126 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-tls-certs\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:17:14.145136 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:14.145135 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-home\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:17:14.145136 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:14.145144 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tqcg\" (UniqueName: \"kubernetes.io/projected/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-kube-api-access-5tqcg\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:17:14.145136 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:14.145153 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1-dshm\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:17:14.745471 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:14.745422 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk" podUID="da54998d-9c50-4743-bc29-271cc2cfadfa" containerName="main" containerID="cri-o://9094a8504c8a4e5e923d43937a742783fa2d3d5e9cac73c48e05b8029c63aca9" gracePeriod=45 Aug 04 11:17:14.877015 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:14.876976 2580 generic.go:356] "Generic (PLEG): container finished" podID="da54998d-9c50-4743-bc29-271cc2cfadfa" containerID="9094a8504c8a4e5e923d43937a742783fa2d3d5e9cac73c48e05b8029c63aca9" exitCode=0 Aug 04 11:17:14.877186 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:14.877023 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk" event={"ID":"da54998d-9c50-4743-bc29-271cc2cfadfa","Type":"ContainerDied","Data":"9094a8504c8a4e5e923d43937a742783fa2d3d5e9cac73c48e05b8029c63aca9"} Aug 04 11:17:14.878529 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:14.878506 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q" event={"ID":"c25cb0e9-c5aa-46b5-be12-4f081cbce3b1","Type":"ContainerDied","Data":"08132b91aaac7565271068518d6c557c9943f58f8ba2d79490c1353cbb65df27"} Aug 04 11:17:14.878683 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:14.878549 2580 scope.go:117] "RemoveContainer" containerID="64e0bdc15588559cf15c88e1b40fd4e9fd2ba34142ed3c7a79fe0cf39f54e2ab" Aug 04 11:17:14.878683 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:14.878555 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q" Aug 04 11:17:14.903478 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:14.903447 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q"] Aug 04 11:17:14.908390 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:14.908357 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-bc4a78/auth-test-model-bc4a78-kserve-5b6b4bf467-x524q"] Aug 04 11:17:15.001696 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:15.001616 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk" Aug 04 11:17:15.099524 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:15.099492 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c25cb0e9-c5aa-46b5-be12-4f081cbce3b1" path="/var/lib/kubelet/pods/c25cb0e9-c5aa-46b5-be12-4f081cbce3b1/volumes" Aug 04 11:17:15.151698 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:15.151663 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da54998d-9c50-4743-bc29-271cc2cfadfa-tmp-dir\") pod \"da54998d-9c50-4743-bc29-271cc2cfadfa\" (UID: \"da54998d-9c50-4743-bc29-271cc2cfadfa\") " Aug 04 11:17:15.151698 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:15.151705 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da54998d-9c50-4743-bc29-271cc2cfadfa-tls-certs\") pod \"da54998d-9c50-4743-bc29-271cc2cfadfa\" (UID: \"da54998d-9c50-4743-bc29-271cc2cfadfa\") " Aug 04 11:17:15.151948 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:15.151746 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da54998d-9c50-4743-bc29-271cc2cfadfa-model-cache\") pod \"da54998d-9c50-4743-bc29-271cc2cfadfa\" (UID: \"da54998d-9c50-4743-bc29-271cc2cfadfa\") " Aug 04 11:17:15.151948 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:15.151776 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da54998d-9c50-4743-bc29-271cc2cfadfa-home\") pod \"da54998d-9c50-4743-bc29-271cc2cfadfa\" (UID: \"da54998d-9c50-4743-bc29-271cc2cfadfa\") " Aug 04 11:17:15.151948 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:15.151811 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjw9r\" (UniqueName: \"kubernetes.io/projected/da54998d-9c50-4743-bc29-271cc2cfadfa-kube-api-access-xjw9r\") pod \"da54998d-9c50-4743-bc29-271cc2cfadfa\" (UID: \"da54998d-9c50-4743-bc29-271cc2cfadfa\") " Aug 04 11:17:15.151948 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:15.151831 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da54998d-9c50-4743-bc29-271cc2cfadfa-dshm\") pod \"da54998d-9c50-4743-bc29-271cc2cfadfa\" (UID: \"da54998d-9c50-4743-bc29-271cc2cfadfa\") " Aug 04 11:17:15.152205 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:15.151997 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da54998d-9c50-4743-bc29-271cc2cfadfa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "da54998d-9c50-4743-bc29-271cc2cfadfa" (UID: "da54998d-9c50-4743-bc29-271cc2cfadfa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:17:15.152205 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:15.152075 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da54998d-9c50-4743-bc29-271cc2cfadfa-model-cache" (OuterVolumeSpecName: "model-cache") pod "da54998d-9c50-4743-bc29-271cc2cfadfa" (UID: "da54998d-9c50-4743-bc29-271cc2cfadfa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:17:15.152205 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:15.152091 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da54998d-9c50-4743-bc29-271cc2cfadfa-home" (OuterVolumeSpecName: "home") pod "da54998d-9c50-4743-bc29-271cc2cfadfa" (UID: "da54998d-9c50-4743-bc29-271cc2cfadfa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:17:15.152205 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:15.152119 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da54998d-9c50-4743-bc29-271cc2cfadfa-tmp-dir\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:17:15.154206 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:15.154178 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da54998d-9c50-4743-bc29-271cc2cfadfa-dshm" (OuterVolumeSpecName: "dshm") pod "da54998d-9c50-4743-bc29-271cc2cfadfa" (UID: "da54998d-9c50-4743-bc29-271cc2cfadfa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:17:15.154329 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:15.154238 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da54998d-9c50-4743-bc29-271cc2cfadfa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "da54998d-9c50-4743-bc29-271cc2cfadfa" (UID: "da54998d-9c50-4743-bc29-271cc2cfadfa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:17:15.154329 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:15.154257 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da54998d-9c50-4743-bc29-271cc2cfadfa-kube-api-access-xjw9r" (OuterVolumeSpecName: "kube-api-access-xjw9r") pod "da54998d-9c50-4743-bc29-271cc2cfadfa" (UID: "da54998d-9c50-4743-bc29-271cc2cfadfa"). InnerVolumeSpecName "kube-api-access-xjw9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:17:15.253070 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:15.252979 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da54998d-9c50-4743-bc29-271cc2cfadfa-tls-certs\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:17:15.253070 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:15.253011 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da54998d-9c50-4743-bc29-271cc2cfadfa-model-cache\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:17:15.253070 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:15.253019 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da54998d-9c50-4743-bc29-271cc2cfadfa-home\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:17:15.253070 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:15.253029 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjw9r\" (UniqueName: \"kubernetes.io/projected/da54998d-9c50-4743-bc29-271cc2cfadfa-kube-api-access-xjw9r\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:17:15.253070 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:15.253038 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da54998d-9c50-4743-bc29-271cc2cfadfa-dshm\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:17:15.884933 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:15.884904 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk" Aug 04 11:17:15.885411 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:15.884937 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk" event={"ID":"da54998d-9c50-4743-bc29-271cc2cfadfa","Type":"ContainerDied","Data":"f485faac45f8d7e7e673070672b11d02d47000c1ae772e36ceb1562391635355"} Aug 04 11:17:15.885411 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:15.884997 2580 scope.go:117] "RemoveContainer" containerID="9094a8504c8a4e5e923d43937a742783fa2d3d5e9cac73c48e05b8029c63aca9" Aug 04 11:17:15.909627 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:15.909600 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk"] Aug 04 11:17:15.912704 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:15.912678 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-460cac/auth-test-model-460cac-kserve-7745b798dc-9m6bk"] Aug 04 11:17:17.099554 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:17.099516 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da54998d-9c50-4743-bc29-271cc2cfadfa" path="/var/lib/kubelet/pods/da54998d-9c50-4743-bc29-271cc2cfadfa/volumes" Aug 04 11:17:19.057117 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.057083 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj"] Aug 04 11:17:19.057514 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.057434 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da54998d-9c50-4743-bc29-271cc2cfadfa" containerName="main" Aug 04 11:17:19.057514 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.057447 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="da54998d-9c50-4743-bc29-271cc2cfadfa" containerName="main" Aug 04 11:17:19.057514 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.057487 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c25cb0e9-c5aa-46b5-be12-4f081cbce3b1" containerName="main" Aug 04 11:17:19.057514 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.057494 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c25cb0e9-c5aa-46b5-be12-4f081cbce3b1" containerName="main" Aug 04 11:17:19.057689 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.057546 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c25cb0e9-c5aa-46b5-be12-4f081cbce3b1" containerName="main" Aug 04 11:17:19.057689 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.057553 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="da54998d-9c50-4743-bc29-271cc2cfadfa" containerName="main" Aug 04 11:17:19.060313 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.060296 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj" Aug 04 11:17:19.065318 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.065299 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-bc4a78\"/\"sub-test-model-bc4a78-kserve-self-signed-certs\"" Aug 04 11:17:19.066757 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.066736 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-bc4a78\"/\"kube-root-ca.crt\"" Aug 04 11:17:19.066874 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.066798 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-bc4a78\"/\"default-dockercfg-xmdqk\"" Aug 04 11:17:19.067259 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.066887 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-bc4a78\"/\"openshift-service-ca.crt\"" Aug 04 11:17:19.080466 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.080429 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj"] Aug 04 11:17:19.190834 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.190793 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17cc3608-2b1a-4382-b717-5a9aabaad502-home\") pod \"sub-test-model-bc4a78-kserve-f54f585c7-2krwj\" (UID: \"17cc3608-2b1a-4382-b717-5a9aabaad502\") " pod="ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj" Aug 04 11:17:19.191051 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.190871 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17cc3608-2b1a-4382-b717-5a9aabaad502-tmp-dir\") pod \"sub-test-model-bc4a78-kserve-f54f585c7-2krwj\" (UID: \"17cc3608-2b1a-4382-b717-5a9aabaad502\") " pod="ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj" Aug 04 11:17:19.191051 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.190975 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmwr9\" (UniqueName: \"kubernetes.io/projected/17cc3608-2b1a-4382-b717-5a9aabaad502-kube-api-access-zmwr9\") pod \"sub-test-model-bc4a78-kserve-f54f585c7-2krwj\" (UID: \"17cc3608-2b1a-4382-b717-5a9aabaad502\") " pod="ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj" Aug 04 11:17:19.191202 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.191074 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17cc3608-2b1a-4382-b717-5a9aabaad502-tls-certs\") pod \"sub-test-model-bc4a78-kserve-f54f585c7-2krwj\" (UID: \"17cc3608-2b1a-4382-b717-5a9aabaad502\") " pod="ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj" Aug 04 11:17:19.191202 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.191108 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17cc3608-2b1a-4382-b717-5a9aabaad502-dshm\") pod \"sub-test-model-bc4a78-kserve-f54f585c7-2krwj\" (UID: \"17cc3608-2b1a-4382-b717-5a9aabaad502\") " pod="ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj" Aug 04 11:17:19.191202 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.191148 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17cc3608-2b1a-4382-b717-5a9aabaad502-model-cache\") pod \"sub-test-model-bc4a78-kserve-f54f585c7-2krwj\" (UID: \"17cc3608-2b1a-4382-b717-5a9aabaad502\") " pod="ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj" Aug 04 11:17:19.292266 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.292232 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17cc3608-2b1a-4382-b717-5a9aabaad502-home\") pod \"sub-test-model-bc4a78-kserve-f54f585c7-2krwj\" (UID: \"17cc3608-2b1a-4382-b717-5a9aabaad502\") " pod="ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj" Aug 04 11:17:19.292266 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.292270 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17cc3608-2b1a-4382-b717-5a9aabaad502-tmp-dir\") pod \"sub-test-model-bc4a78-kserve-f54f585c7-2krwj\" (UID: \"17cc3608-2b1a-4382-b717-5a9aabaad502\") " pod="ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj" Aug 04 11:17:19.292519 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.292307 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmwr9\" (UniqueName: \"kubernetes.io/projected/17cc3608-2b1a-4382-b717-5a9aabaad502-kube-api-access-zmwr9\") pod \"sub-test-model-bc4a78-kserve-f54f585c7-2krwj\" (UID: \"17cc3608-2b1a-4382-b717-5a9aabaad502\") " pod="ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj" Aug 04 11:17:19.292519 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.292342 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17cc3608-2b1a-4382-b717-5a9aabaad502-tls-certs\") pod \"sub-test-model-bc4a78-kserve-f54f585c7-2krwj\" (UID: \"17cc3608-2b1a-4382-b717-5a9aabaad502\") " pod="ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj" Aug 04 11:17:19.292519 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.292359 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17cc3608-2b1a-4382-b717-5a9aabaad502-dshm\") pod \"sub-test-model-bc4a78-kserve-f54f585c7-2krwj\" (UID: \"17cc3608-2b1a-4382-b717-5a9aabaad502\") " pod="ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj" Aug 04 11:17:19.292519 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.292388 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17cc3608-2b1a-4382-b717-5a9aabaad502-model-cache\") pod \"sub-test-model-bc4a78-kserve-f54f585c7-2krwj\" (UID: \"17cc3608-2b1a-4382-b717-5a9aabaad502\") " pod="ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj" Aug 04 11:17:19.292756 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.292729 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17cc3608-2b1a-4382-b717-5a9aabaad502-home\") pod \"sub-test-model-bc4a78-kserve-f54f585c7-2krwj\" (UID: \"17cc3608-2b1a-4382-b717-5a9aabaad502\") " pod="ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj" Aug 04 11:17:19.292846 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.292779 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17cc3608-2b1a-4382-b717-5a9aabaad502-model-cache\") pod \"sub-test-model-bc4a78-kserve-f54f585c7-2krwj\" (UID: \"17cc3608-2b1a-4382-b717-5a9aabaad502\") " pod="ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj" Aug 04 11:17:19.292846 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.292824 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17cc3608-2b1a-4382-b717-5a9aabaad502-tmp-dir\") pod \"sub-test-model-bc4a78-kserve-f54f585c7-2krwj\" (UID: \"17cc3608-2b1a-4382-b717-5a9aabaad502\") " pod="ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj" Aug 04 11:17:19.294847 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.294827 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17cc3608-2b1a-4382-b717-5a9aabaad502-dshm\") pod \"sub-test-model-bc4a78-kserve-f54f585c7-2krwj\" (UID: \"17cc3608-2b1a-4382-b717-5a9aabaad502\") " pod="ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj" Aug 04 11:17:19.295212 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.295188 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17cc3608-2b1a-4382-b717-5a9aabaad502-tls-certs\") pod \"sub-test-model-bc4a78-kserve-f54f585c7-2krwj\" (UID: \"17cc3608-2b1a-4382-b717-5a9aabaad502\") " pod="ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj" Aug 04 11:17:19.304526 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.304495 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmwr9\" (UniqueName: \"kubernetes.io/projected/17cc3608-2b1a-4382-b717-5a9aabaad502-kube-api-access-zmwr9\") pod \"sub-test-model-bc4a78-kserve-f54f585c7-2krwj\" (UID: \"17cc3608-2b1a-4382-b717-5a9aabaad502\") " pod="ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj" Aug 04 11:17:19.370074 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.370044 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj" Aug 04 11:17:19.534442 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.534417 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj"] Aug 04 11:17:19.536461 ip-10-0-142-113 kubenswrapper[2580]: W0804 11:17:19.536431 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17cc3608_2b1a_4382_b717_5a9aabaad502.slice/crio-fcc3b2e2d16c44b0ae28fcbd250373b007fe6b8e1f99b1fbef1fe97cbbd035ce WatchSource:0}: Error finding container fcc3b2e2d16c44b0ae28fcbd250373b007fe6b8e1f99b1fbef1fe97cbbd035ce: Status 404 returned error can't find the container with id fcc3b2e2d16c44b0ae28fcbd250373b007fe6b8e1f99b1fbef1fe97cbbd035ce Aug 04 11:17:19.909871 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.909828 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj" event={"ID":"17cc3608-2b1a-4382-b717-5a9aabaad502","Type":"ContainerStarted","Data":"820f96e2dbabb932e78898976431a1cd6c414feb29551e8454b0467c86c9b422"} Aug 04 11:17:19.910046 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:19.909884 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj" event={"ID":"17cc3608-2b1a-4382-b717-5a9aabaad502","Type":"ContainerStarted","Data":"fcc3b2e2d16c44b0ae28fcbd250373b007fe6b8e1f99b1fbef1fe97cbbd035ce"} Aug 04 11:17:21.673638 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:21.673578 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj" podStartSLOduration=2.673541471 podStartE2EDuration="2.673541471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:17:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:17:19.950532317 +0000 UTC m=+2979.371419727" watchObservedRunningTime="2026-08-04 11:17:21.673541471 +0000 UTC m=+2981.094428882" Aug 04 11:17:21.674202 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:21.674180 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:17:29.370425 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:29.370385 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj" Aug 04 11:17:29.370901 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:29.370486 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj" Aug 04 11:17:29.383280 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:29.383250 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj" Aug 04 11:17:29.971111 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:29.971083 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj" Aug 04 11:17:41.255026 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:41.254999 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tb4zk_c5922147-cb42-4f62-9880-8deaa51a4b0f/ovn-acl-logging/0.log" Aug 04 11:17:41.258441 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:41.258421 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tb4zk_c5922147-cb42-4f62-9880-8deaa51a4b0f/ovn-acl-logging/0.log" Aug 04 11:17:42.248669 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:42.248628 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5"] Aug 04 11:17:42.253725 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:42.253704 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5" Aug 04 11:17:42.258434 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:42.258405 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-460cac\"/\"openshift-service-ca.crt\"" Aug 04 11:17:42.258434 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:42.258425 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-460cac\"/\"default-dockercfg-c7lzs\"" Aug 04 11:17:42.258851 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:42.258407 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-460cac\"/\"kube-root-ca.crt\"" Aug 04 11:17:42.258851 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:42.258428 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-460cac\"/\"sub-test-model-460cac-kserve-self-signed-certs\"" Aug 04 11:17:42.264260 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:42.264234 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5"] Aug 04 11:17:42.293575 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:42.293524 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99389820-45f5-43d7-92f8-1896bd1b6953-tls-certs\") pod \"sub-test-model-460cac-kserve-7cb75c75b9-zj5s5\" (UID: \"99389820-45f5-43d7-92f8-1896bd1b6953\") " pod="ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5" Aug 04 11:17:42.293736 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:42.293642 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99389820-45f5-43d7-92f8-1896bd1b6953-home\") pod \"sub-test-model-460cac-kserve-7cb75c75b9-zj5s5\" (UID: \"99389820-45f5-43d7-92f8-1896bd1b6953\") " pod="ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5" Aug 04 11:17:42.293736 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:42.293719 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7kx2\" (UniqueName: \"kubernetes.io/projected/99389820-45f5-43d7-92f8-1896bd1b6953-kube-api-access-f7kx2\") pod \"sub-test-model-460cac-kserve-7cb75c75b9-zj5s5\" (UID: \"99389820-45f5-43d7-92f8-1896bd1b6953\") " pod="ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5" Aug 04 11:17:42.293811 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:42.293755 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99389820-45f5-43d7-92f8-1896bd1b6953-tmp-dir\") pod \"sub-test-model-460cac-kserve-7cb75c75b9-zj5s5\" (UID: \"99389820-45f5-43d7-92f8-1896bd1b6953\") " pod="ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5" Aug 04 11:17:42.293811 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:42.293787 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99389820-45f5-43d7-92f8-1896bd1b6953-dshm\") pod \"sub-test-model-460cac-kserve-7cb75c75b9-zj5s5\" (UID: \"99389820-45f5-43d7-92f8-1896bd1b6953\") " pod="ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5" Aug 04 11:17:42.293881 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:42.293854 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99389820-45f5-43d7-92f8-1896bd1b6953-model-cache\") pod \"sub-test-model-460cac-kserve-7cb75c75b9-zj5s5\" (UID: \"99389820-45f5-43d7-92f8-1896bd1b6953\") " pod="ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5" Aug 04 11:17:42.395170 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:42.395127 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7kx2\" (UniqueName: \"kubernetes.io/projected/99389820-45f5-43d7-92f8-1896bd1b6953-kube-api-access-f7kx2\") pod \"sub-test-model-460cac-kserve-7cb75c75b9-zj5s5\" (UID: \"99389820-45f5-43d7-92f8-1896bd1b6953\") " pod="ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5" Aug 04 11:17:42.395170 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:42.395174 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99389820-45f5-43d7-92f8-1896bd1b6953-tmp-dir\") pod \"sub-test-model-460cac-kserve-7cb75c75b9-zj5s5\" (UID: \"99389820-45f5-43d7-92f8-1896bd1b6953\") " pod="ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5" Aug 04 11:17:42.395423 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:42.395291 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99389820-45f5-43d7-92f8-1896bd1b6953-dshm\") pod \"sub-test-model-460cac-kserve-7cb75c75b9-zj5s5\" (UID: \"99389820-45f5-43d7-92f8-1896bd1b6953\") " pod="ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5" Aug 04 11:17:42.395423 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:42.395336 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99389820-45f5-43d7-92f8-1896bd1b6953-model-cache\") pod \"sub-test-model-460cac-kserve-7cb75c75b9-zj5s5\" (UID: \"99389820-45f5-43d7-92f8-1896bd1b6953\") " pod="ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5" Aug 04 11:17:42.395423 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:42.395419 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99389820-45f5-43d7-92f8-1896bd1b6953-tls-certs\") pod \"sub-test-model-460cac-kserve-7cb75c75b9-zj5s5\" (UID: \"99389820-45f5-43d7-92f8-1896bd1b6953\") " pod="ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5" Aug 04 11:17:42.395657 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:42.395482 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99389820-45f5-43d7-92f8-1896bd1b6953-home\") pod \"sub-test-model-460cac-kserve-7cb75c75b9-zj5s5\" (UID: \"99389820-45f5-43d7-92f8-1896bd1b6953\") " pod="ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5" Aug 04 11:17:42.395657 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:42.395608 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99389820-45f5-43d7-92f8-1896bd1b6953-tmp-dir\") pod \"sub-test-model-460cac-kserve-7cb75c75b9-zj5s5\" (UID: \"99389820-45f5-43d7-92f8-1896bd1b6953\") " pod="ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5" Aug 04 11:17:42.395776 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:42.395751 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99389820-45f5-43d7-92f8-1896bd1b6953-model-cache\") pod \"sub-test-model-460cac-kserve-7cb75c75b9-zj5s5\" (UID: \"99389820-45f5-43d7-92f8-1896bd1b6953\") " pod="ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5" Aug 04 11:17:42.395817 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:42.395773 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99389820-45f5-43d7-92f8-1896bd1b6953-home\") pod \"sub-test-model-460cac-kserve-7cb75c75b9-zj5s5\" (UID: \"99389820-45f5-43d7-92f8-1896bd1b6953\") " pod="ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5" Aug 04 11:17:42.397838 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:42.397808 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99389820-45f5-43d7-92f8-1896bd1b6953-dshm\") pod \"sub-test-model-460cac-kserve-7cb75c75b9-zj5s5\" (UID: \"99389820-45f5-43d7-92f8-1896bd1b6953\") " pod="ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5" Aug 04 11:17:42.398220 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:42.398198 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99389820-45f5-43d7-92f8-1896bd1b6953-tls-certs\") pod \"sub-test-model-460cac-kserve-7cb75c75b9-zj5s5\" (UID: \"99389820-45f5-43d7-92f8-1896bd1b6953\") " pod="ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5" Aug 04 11:17:42.406991 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:42.406926 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7kx2\" (UniqueName: \"kubernetes.io/projected/99389820-45f5-43d7-92f8-1896bd1b6953-kube-api-access-f7kx2\") pod \"sub-test-model-460cac-kserve-7cb75c75b9-zj5s5\" (UID: \"99389820-45f5-43d7-92f8-1896bd1b6953\") " pod="ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5" Aug 04 11:17:42.564083 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:42.563997 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5" Aug 04 11:17:42.710246 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:42.710207 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5"] Aug 04 11:17:42.713329 ip-10-0-142-113 kubenswrapper[2580]: W0804 11:17:42.713300 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99389820_45f5_43d7_92f8_1896bd1b6953.slice/crio-6a66f9f1b23c5369c2cf3587baaf8c9a129f158e4ec4394da9d783d847bb2288 WatchSource:0}: Error finding container 6a66f9f1b23c5369c2cf3587baaf8c9a129f158e4ec4394da9d783d847bb2288: Status 404 returned error can't find the container with id 6a66f9f1b23c5369c2cf3587baaf8c9a129f158e4ec4394da9d783d847bb2288 Aug 04 11:17:43.026032 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:43.025994 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5" event={"ID":"99389820-45f5-43d7-92f8-1896bd1b6953","Type":"ContainerStarted","Data":"abbf3ec591511c5c1d28481441e7288bf726b85daabba3ca5efc16dc2b16298a"} Aug 04 11:17:43.026032 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:43.026037 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5" event={"ID":"99389820-45f5-43d7-92f8-1896bd1b6953","Type":"ContainerStarted","Data":"6a66f9f1b23c5369c2cf3587baaf8c9a129f158e4ec4394da9d783d847bb2288"} Aug 04 11:17:43.048838 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:43.048791 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5" podStartSLOduration=1.048774357 podStartE2EDuration="1.048774357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:17:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:17:43.04866867 +0000 UTC m=+3002.469556078" watchObservedRunningTime="2026-08-04 11:17:43.048774357 +0000 UTC m=+3002.469661768" Aug 04 11:17:45.143799 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:45.143764 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:17:52.564234 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:52.564194 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5" Aug 04 11:17:52.564724 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:52.564279 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5" Aug 04 11:17:52.576970 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:52.576938 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5" Aug 04 11:17:53.087980 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:17:53.087951 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5" Aug 04 11:18:22.848715 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:18:22.848679 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:18:29.937112 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:18:29.937075 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:18:41.407863 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:18:41.407834 2580 scope.go:117] "RemoveContainer" containerID="753116c0ecbcdae9656ebb8d16f68bd3d2548feddbecd71c2869c1c195ac67f5" Aug 04 11:18:47.739711 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:18:47.739672 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:18:51.987388 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:18:51.987352 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj"] Aug 04 11:18:53.045846 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:18:53.045814 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5"] Aug 04 11:18:58.133677 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:18:58.133637 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:19:04.842908 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:04.842873 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8"] Aug 04 11:19:04.846267 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:04.846244 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8" Aug 04 11:19:04.849369 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:04.849338 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-bc4a78\"/\"rate-test-model-bc4a78-kserve-self-signed-certs\"" Aug 04 11:19:04.857425 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:04.857399 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8"] Aug 04 11:19:04.960869 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:04.960827 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5a82d0c-de1c-4011-8d2d-8fd665006701-tls-certs\") pod \"rate-test-model-bc4a78-kserve-598c645b79-cq2d8\" (UID: \"b5a82d0c-de1c-4011-8d2d-8fd665006701\") " pod="ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8" Aug 04 11:19:04.961162 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:04.961137 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5a82d0c-de1c-4011-8d2d-8fd665006701-tmp-dir\") pod \"rate-test-model-bc4a78-kserve-598c645b79-cq2d8\" (UID: \"b5a82d0c-de1c-4011-8d2d-8fd665006701\") " pod="ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8" Aug 04 11:19:04.961304 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:04.961282 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slswj\" (UniqueName: \"kubernetes.io/projected/b5a82d0c-de1c-4011-8d2d-8fd665006701-kube-api-access-slswj\") pod \"rate-test-model-bc4a78-kserve-598c645b79-cq2d8\" (UID: \"b5a82d0c-de1c-4011-8d2d-8fd665006701\") " pod="ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8" Aug 04 11:19:04.961447 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:04.961423 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5a82d0c-de1c-4011-8d2d-8fd665006701-dshm\") pod \"rate-test-model-bc4a78-kserve-598c645b79-cq2d8\" (UID: \"b5a82d0c-de1c-4011-8d2d-8fd665006701\") " pod="ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8" Aug 04 11:19:04.961641 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:04.961622 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5a82d0c-de1c-4011-8d2d-8fd665006701-model-cache\") pod \"rate-test-model-bc4a78-kserve-598c645b79-cq2d8\" (UID: \"b5a82d0c-de1c-4011-8d2d-8fd665006701\") " pod="ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8" Aug 04 11:19:04.961795 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:04.961768 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5a82d0c-de1c-4011-8d2d-8fd665006701-home\") pod \"rate-test-model-bc4a78-kserve-598c645b79-cq2d8\" (UID: \"b5a82d0c-de1c-4011-8d2d-8fd665006701\") " pod="ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8" Aug 04 11:19:05.062788 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:05.062739 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5a82d0c-de1c-4011-8d2d-8fd665006701-model-cache\") pod \"rate-test-model-bc4a78-kserve-598c645b79-cq2d8\" (UID: \"b5a82d0c-de1c-4011-8d2d-8fd665006701\") " pod="ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8" Aug 04 11:19:05.062788 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:05.062791 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5a82d0c-de1c-4011-8d2d-8fd665006701-home\") pod \"rate-test-model-bc4a78-kserve-598c645b79-cq2d8\" (UID: \"b5a82d0c-de1c-4011-8d2d-8fd665006701\") " pod="ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8" Aug 04 11:19:05.063059 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:05.062871 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5a82d0c-de1c-4011-8d2d-8fd665006701-tls-certs\") pod \"rate-test-model-bc4a78-kserve-598c645b79-cq2d8\" (UID: \"b5a82d0c-de1c-4011-8d2d-8fd665006701\") " pod="ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8" Aug 04 11:19:05.063059 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:05.062909 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5a82d0c-de1c-4011-8d2d-8fd665006701-tmp-dir\") pod \"rate-test-model-bc4a78-kserve-598c645b79-cq2d8\" (UID: \"b5a82d0c-de1c-4011-8d2d-8fd665006701\") " pod="ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8" Aug 04 11:19:05.063059 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:05.062932 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slswj\" (UniqueName: \"kubernetes.io/projected/b5a82d0c-de1c-4011-8d2d-8fd665006701-kube-api-access-slswj\") pod \"rate-test-model-bc4a78-kserve-598c645b79-cq2d8\" (UID: \"b5a82d0c-de1c-4011-8d2d-8fd665006701\") " pod="ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8" Aug 04 11:19:05.063059 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:05.062967 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5a82d0c-de1c-4011-8d2d-8fd665006701-dshm\") pod \"rate-test-model-bc4a78-kserve-598c645b79-cq2d8\" (UID: \"b5a82d0c-de1c-4011-8d2d-8fd665006701\") " pod="ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8" Aug 04 11:19:05.063268 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:05.063212 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5a82d0c-de1c-4011-8d2d-8fd665006701-model-cache\") pod \"rate-test-model-bc4a78-kserve-598c645b79-cq2d8\" (UID: \"b5a82d0c-de1c-4011-8d2d-8fd665006701\") " pod="ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8" Aug 04 11:19:05.063330 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:05.063249 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5a82d0c-de1c-4011-8d2d-8fd665006701-home\") pod \"rate-test-model-bc4a78-kserve-598c645b79-cq2d8\" (UID: \"b5a82d0c-de1c-4011-8d2d-8fd665006701\") " pod="ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8" Aug 04 11:19:05.063330 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:05.063324 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5a82d0c-de1c-4011-8d2d-8fd665006701-tmp-dir\") pod \"rate-test-model-bc4a78-kserve-598c645b79-cq2d8\" (UID: \"b5a82d0c-de1c-4011-8d2d-8fd665006701\") " pod="ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8" Aug 04 11:19:05.065302 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:05.065281 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5a82d0c-de1c-4011-8d2d-8fd665006701-dshm\") pod \"rate-test-model-bc4a78-kserve-598c645b79-cq2d8\" (UID: \"b5a82d0c-de1c-4011-8d2d-8fd665006701\") " pod="ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8" Aug 04 11:19:05.065628 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:05.065603 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5a82d0c-de1c-4011-8d2d-8fd665006701-tls-certs\") pod \"rate-test-model-bc4a78-kserve-598c645b79-cq2d8\" (UID: \"b5a82d0c-de1c-4011-8d2d-8fd665006701\") " pod="ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8" Aug 04 11:19:05.072036 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:05.072013 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slswj\" (UniqueName: \"kubernetes.io/projected/b5a82d0c-de1c-4011-8d2d-8fd665006701-kube-api-access-slswj\") pod \"rate-test-model-bc4a78-kserve-598c645b79-cq2d8\" (UID: \"b5a82d0c-de1c-4011-8d2d-8fd665006701\") " pod="ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8" Aug 04 11:19:05.155459 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:05.155420 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8" Aug 04 11:19:05.299520 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:05.299495 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8"] Aug 04 11:19:05.302284 ip-10-0-142-113 kubenswrapper[2580]: W0804 11:19:05.302253 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5a82d0c_de1c_4011_8d2d_8fd665006701.slice/crio-c345244af077d0a985eedd9c2b78480db329a3a1c51a73f1c0b8c7387dda14ae WatchSource:0}: Error finding container c345244af077d0a985eedd9c2b78480db329a3a1c51a73f1c0b8c7387dda14ae: Status 404 returned error can't find the container with id c345244af077d0a985eedd9c2b78480db329a3a1c51a73f1c0b8c7387dda14ae Aug 04 11:19:05.449784 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:05.449693 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8" event={"ID":"b5a82d0c-de1c-4011-8d2d-8fd665006701","Type":"ContainerStarted","Data":"1ac11aac39eb48fe9871b9a5450e867147f5df0c88296e40a437e88ab14850d1"} Aug 04 11:19:05.449784 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:05.449738 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8" event={"ID":"b5a82d0c-de1c-4011-8d2d-8fd665006701","Type":"ContainerStarted","Data":"c345244af077d0a985eedd9c2b78480db329a3a1c51a73f1c0b8c7387dda14ae"} Aug 04 11:19:05.471932 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:05.471876 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8" podStartSLOduration=1.47186176 podStartE2EDuration="1.47186176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:19:05.470491618 +0000 UTC m=+3084.891379031" watchObservedRunningTime="2026-08-04 11:19:05.47186176 +0000 UTC m=+3084.892749170" Aug 04 11:19:07.006817 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:07.006778 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj" podUID="17cc3608-2b1a-4382-b717-5a9aabaad502" containerName="main" containerID="cri-o://820f96e2dbabb932e78898976431a1cd6c414feb29551e8454b0467c86c9b422" gracePeriod=45 Aug 04 11:19:07.261720 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:07.261654 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj" Aug 04 11:19:07.383656 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:07.383623 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17cc3608-2b1a-4382-b717-5a9aabaad502-home\") pod \"17cc3608-2b1a-4382-b717-5a9aabaad502\" (UID: \"17cc3608-2b1a-4382-b717-5a9aabaad502\") " Aug 04 11:19:07.383850 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:07.383706 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmwr9\" (UniqueName: \"kubernetes.io/projected/17cc3608-2b1a-4382-b717-5a9aabaad502-kube-api-access-zmwr9\") pod \"17cc3608-2b1a-4382-b717-5a9aabaad502\" (UID: \"17cc3608-2b1a-4382-b717-5a9aabaad502\") " Aug 04 11:19:07.383850 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:07.383726 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17cc3608-2b1a-4382-b717-5a9aabaad502-tmp-dir\") pod \"17cc3608-2b1a-4382-b717-5a9aabaad502\" (UID: \"17cc3608-2b1a-4382-b717-5a9aabaad502\") " Aug 04 11:19:07.383850 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:07.383769 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17cc3608-2b1a-4382-b717-5a9aabaad502-model-cache\") pod \"17cc3608-2b1a-4382-b717-5a9aabaad502\" (UID: \"17cc3608-2b1a-4382-b717-5a9aabaad502\") " Aug 04 11:19:07.383850 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:07.383815 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17cc3608-2b1a-4382-b717-5a9aabaad502-tls-certs\") pod \"17cc3608-2b1a-4382-b717-5a9aabaad502\" (UID: \"17cc3608-2b1a-4382-b717-5a9aabaad502\") " Aug 04 11:19:07.383850 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:07.383839 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17cc3608-2b1a-4382-b717-5a9aabaad502-dshm\") pod \"17cc3608-2b1a-4382-b717-5a9aabaad502\" (UID: \"17cc3608-2b1a-4382-b717-5a9aabaad502\") " Aug 04 11:19:07.384125 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:07.383952 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17cc3608-2b1a-4382-b717-5a9aabaad502-home" (OuterVolumeSpecName: "home") pod "17cc3608-2b1a-4382-b717-5a9aabaad502" (UID: "17cc3608-2b1a-4382-b717-5a9aabaad502"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:19:07.384125 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:07.384022 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17cc3608-2b1a-4382-b717-5a9aabaad502-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "17cc3608-2b1a-4382-b717-5a9aabaad502" (UID: "17cc3608-2b1a-4382-b717-5a9aabaad502"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:19:07.384266 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:07.384246 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17cc3608-2b1a-4382-b717-5a9aabaad502-home\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:19:07.384320 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:07.384272 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17cc3608-2b1a-4382-b717-5a9aabaad502-tmp-dir\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:19:07.384379 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:07.384362 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17cc3608-2b1a-4382-b717-5a9aabaad502-model-cache" (OuterVolumeSpecName: "model-cache") pod "17cc3608-2b1a-4382-b717-5a9aabaad502" (UID: "17cc3608-2b1a-4382-b717-5a9aabaad502"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:19:07.386287 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:07.386245 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17cc3608-2b1a-4382-b717-5a9aabaad502-kube-api-access-zmwr9" (OuterVolumeSpecName: "kube-api-access-zmwr9") pod "17cc3608-2b1a-4382-b717-5a9aabaad502" (UID: "17cc3608-2b1a-4382-b717-5a9aabaad502"). InnerVolumeSpecName "kube-api-access-zmwr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:19:07.386717 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:07.386693 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17cc3608-2b1a-4382-b717-5a9aabaad502-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "17cc3608-2b1a-4382-b717-5a9aabaad502" (UID: "17cc3608-2b1a-4382-b717-5a9aabaad502"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:19:07.386793 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:07.386719 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17cc3608-2b1a-4382-b717-5a9aabaad502-dshm" (OuterVolumeSpecName: "dshm") pod "17cc3608-2b1a-4382-b717-5a9aabaad502" (UID: "17cc3608-2b1a-4382-b717-5a9aabaad502"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:19:07.426699 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:07.426652 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:19:07.461621 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:07.461581 2580 generic.go:356] "Generic (PLEG): container finished" podID="17cc3608-2b1a-4382-b717-5a9aabaad502" containerID="820f96e2dbabb932e78898976431a1cd6c414feb29551e8454b0467c86c9b422" exitCode=0 Aug 04 11:19:07.461621 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:07.461619 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj" Aug 04 11:19:07.461823 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:07.461658 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj" event={"ID":"17cc3608-2b1a-4382-b717-5a9aabaad502","Type":"ContainerDied","Data":"820f96e2dbabb932e78898976431a1cd6c414feb29551e8454b0467c86c9b422"} Aug 04 11:19:07.461823 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:07.461704 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj" event={"ID":"17cc3608-2b1a-4382-b717-5a9aabaad502","Type":"ContainerDied","Data":"fcc3b2e2d16c44b0ae28fcbd250373b007fe6b8e1f99b1fbef1fe97cbbd035ce"} Aug 04 11:19:07.461823 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:07.461727 2580 scope.go:117] "RemoveContainer" containerID="820f96e2dbabb932e78898976431a1cd6c414feb29551e8454b0467c86c9b422" Aug 04 11:19:07.471653 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:07.471631 2580 scope.go:117] "RemoveContainer" containerID="820f96e2dbabb932e78898976431a1cd6c414feb29551e8454b0467c86c9b422" Aug 04 11:19:07.471966 ip-10-0-142-113 kubenswrapper[2580]: E0804 11:19:07.471948 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"820f96e2dbabb932e78898976431a1cd6c414feb29551e8454b0467c86c9b422\": container with ID starting with 820f96e2dbabb932e78898976431a1cd6c414feb29551e8454b0467c86c9b422 not found: ID does not exist" containerID="820f96e2dbabb932e78898976431a1cd6c414feb29551e8454b0467c86c9b422" Aug 04 11:19:07.472015 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:07.471977 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"820f96e2dbabb932e78898976431a1cd6c414feb29551e8454b0467c86c9b422"} err="failed to get container status \"820f96e2dbabb932e78898976431a1cd6c414feb29551e8454b0467c86c9b422\": rpc error: code = NotFound desc = could not find container \"820f96e2dbabb932e78898976431a1cd6c414feb29551e8454b0467c86c9b422\": container with ID starting with 820f96e2dbabb932e78898976431a1cd6c414feb29551e8454b0467c86c9b422 not found: ID does not exist" Aug 04 11:19:07.485038 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:07.485007 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmwr9\" (UniqueName: \"kubernetes.io/projected/17cc3608-2b1a-4382-b717-5a9aabaad502-kube-api-access-zmwr9\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:19:07.485038 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:07.485035 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17cc3608-2b1a-4382-b717-5a9aabaad502-model-cache\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:19:07.485298 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:07.485051 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17cc3608-2b1a-4382-b717-5a9aabaad502-tls-certs\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:19:07.485298 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:07.485062 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17cc3608-2b1a-4382-b717-5a9aabaad502-dshm\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:19:07.485656 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:07.485609 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj"] Aug 04 11:19:07.487998 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:07.487974 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-bc4a78/sub-test-model-bc4a78-kserve-f54f585c7-2krwj"] Aug 04 11:19:08.065215 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:08.065170 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5" podUID="99389820-45f5-43d7-92f8-1896bd1b6953" containerName="main" containerID="cri-o://abbf3ec591511c5c1d28481441e7288bf726b85daabba3ca5efc16dc2b16298a" gracePeriod=45 Aug 04 11:19:08.329474 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:08.329399 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5" Aug 04 11:19:08.393703 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:08.393659 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99389820-45f5-43d7-92f8-1896bd1b6953-home\") pod \"99389820-45f5-43d7-92f8-1896bd1b6953\" (UID: \"99389820-45f5-43d7-92f8-1896bd1b6953\") " Aug 04 11:19:08.393703 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:08.393714 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99389820-45f5-43d7-92f8-1896bd1b6953-dshm\") pod \"99389820-45f5-43d7-92f8-1896bd1b6953\" (UID: \"99389820-45f5-43d7-92f8-1896bd1b6953\") " Aug 04 11:19:08.393963 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:08.393755 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99389820-45f5-43d7-92f8-1896bd1b6953-tmp-dir\") pod \"99389820-45f5-43d7-92f8-1896bd1b6953\" (UID: \"99389820-45f5-43d7-92f8-1896bd1b6953\") " Aug 04 11:19:08.393963 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:08.393853 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7kx2\" (UniqueName: \"kubernetes.io/projected/99389820-45f5-43d7-92f8-1896bd1b6953-kube-api-access-f7kx2\") pod \"99389820-45f5-43d7-92f8-1896bd1b6953\" (UID: \"99389820-45f5-43d7-92f8-1896bd1b6953\") " Aug 04 11:19:08.393963 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:08.393935 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99389820-45f5-43d7-92f8-1896bd1b6953-tls-certs\") pod \"99389820-45f5-43d7-92f8-1896bd1b6953\" (UID: \"99389820-45f5-43d7-92f8-1896bd1b6953\") " Aug 04 11:19:08.394125 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:08.393973 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99389820-45f5-43d7-92f8-1896bd1b6953-model-cache\") pod \"99389820-45f5-43d7-92f8-1896bd1b6953\" (UID: \"99389820-45f5-43d7-92f8-1896bd1b6953\") " Aug 04 11:19:08.394125 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:08.393933 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99389820-45f5-43d7-92f8-1896bd1b6953-home" (OuterVolumeSpecName: "home") pod "99389820-45f5-43d7-92f8-1896bd1b6953" (UID: "99389820-45f5-43d7-92f8-1896bd1b6953"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:19:08.394125 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:08.394063 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99389820-45f5-43d7-92f8-1896bd1b6953-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "99389820-45f5-43d7-92f8-1896bd1b6953" (UID: "99389820-45f5-43d7-92f8-1896bd1b6953"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:19:08.394299 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:08.394278 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99389820-45f5-43d7-92f8-1896bd1b6953-tmp-dir\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:19:08.394353 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:08.394300 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99389820-45f5-43d7-92f8-1896bd1b6953-home\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:19:08.394353 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:08.394295 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99389820-45f5-43d7-92f8-1896bd1b6953-model-cache" (OuterVolumeSpecName: "model-cache") pod "99389820-45f5-43d7-92f8-1896bd1b6953" (UID: "99389820-45f5-43d7-92f8-1896bd1b6953"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:19:08.396237 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:08.396203 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99389820-45f5-43d7-92f8-1896bd1b6953-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "99389820-45f5-43d7-92f8-1896bd1b6953" (UID: "99389820-45f5-43d7-92f8-1896bd1b6953"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:19:08.396490 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:08.396472 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99389820-45f5-43d7-92f8-1896bd1b6953-dshm" (OuterVolumeSpecName: "dshm") pod "99389820-45f5-43d7-92f8-1896bd1b6953" (UID: "99389820-45f5-43d7-92f8-1896bd1b6953"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:19:08.396545 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:08.396521 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99389820-45f5-43d7-92f8-1896bd1b6953-kube-api-access-f7kx2" (OuterVolumeSpecName: "kube-api-access-f7kx2") pod "99389820-45f5-43d7-92f8-1896bd1b6953" (UID: "99389820-45f5-43d7-92f8-1896bd1b6953"). InnerVolumeSpecName "kube-api-access-f7kx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:19:08.467856 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:08.467822 2580 generic.go:356] "Generic (PLEG): container finished" podID="99389820-45f5-43d7-92f8-1896bd1b6953" containerID="abbf3ec591511c5c1d28481441e7288bf726b85daabba3ca5efc16dc2b16298a" exitCode=0 Aug 04 11:19:08.468072 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:08.467862 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5" Aug 04 11:19:08.468072 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:08.467884 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5" event={"ID":"99389820-45f5-43d7-92f8-1896bd1b6953","Type":"ContainerDied","Data":"abbf3ec591511c5c1d28481441e7288bf726b85daabba3ca5efc16dc2b16298a"} Aug 04 11:19:08.468072 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:08.467925 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5" event={"ID":"99389820-45f5-43d7-92f8-1896bd1b6953","Type":"ContainerDied","Data":"6a66f9f1b23c5369c2cf3587baaf8c9a129f158e4ec4394da9d783d847bb2288"} Aug 04 11:19:08.468072 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:08.467944 2580 scope.go:117] "RemoveContainer" containerID="abbf3ec591511c5c1d28481441e7288bf726b85daabba3ca5efc16dc2b16298a" Aug 04 11:19:08.477902 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:08.477867 2580 scope.go:117] "RemoveContainer" containerID="abbf3ec591511c5c1d28481441e7288bf726b85daabba3ca5efc16dc2b16298a" Aug 04 11:19:08.478199 ip-10-0-142-113 kubenswrapper[2580]: E0804 11:19:08.478175 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abbf3ec591511c5c1d28481441e7288bf726b85daabba3ca5efc16dc2b16298a\": container with ID starting with abbf3ec591511c5c1d28481441e7288bf726b85daabba3ca5efc16dc2b16298a not found: ID does not exist" containerID="abbf3ec591511c5c1d28481441e7288bf726b85daabba3ca5efc16dc2b16298a" Aug 04 11:19:08.478300 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:08.478208 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abbf3ec591511c5c1d28481441e7288bf726b85daabba3ca5efc16dc2b16298a"} err="failed to get container status \"abbf3ec591511c5c1d28481441e7288bf726b85daabba3ca5efc16dc2b16298a\": rpc error: code = NotFound desc = could not find container \"abbf3ec591511c5c1d28481441e7288bf726b85daabba3ca5efc16dc2b16298a\": container with ID starting with abbf3ec591511c5c1d28481441e7288bf726b85daabba3ca5efc16dc2b16298a not found: ID does not exist" Aug 04 11:19:08.492360 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:08.492329 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5"] Aug 04 11:19:08.494916 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:08.494890 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7kx2\" (UniqueName: \"kubernetes.io/projected/99389820-45f5-43d7-92f8-1896bd1b6953-kube-api-access-f7kx2\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:19:08.494916 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:08.494917 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99389820-45f5-43d7-92f8-1896bd1b6953-tls-certs\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:19:08.495118 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:08.494927 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99389820-45f5-43d7-92f8-1896bd1b6953-model-cache\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:19:08.495118 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:08.494935 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99389820-45f5-43d7-92f8-1896bd1b6953-dshm\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:19:08.496100 ip-10-0-142-113 kubenswrapper[2580]: E0804 11:19:08.496073 2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99389820_45f5_43d7_92f8_1896bd1b6953.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99389820_45f5_43d7_92f8_1896bd1b6953.slice/crio-6a66f9f1b23c5369c2cf3587baaf8c9a129f158e4ec4394da9d783d847bb2288\": RecentStats: unable to find data in memory cache]" Aug 04 11:19:08.497152 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:08.497133 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-460cac/sub-test-model-460cac-kserve-7cb75c75b9-zj5s5"] Aug 04 11:19:09.100045 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:09.099963 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17cc3608-2b1a-4382-b717-5a9aabaad502" path="/var/lib/kubelet/pods/17cc3608-2b1a-4382-b717-5a9aabaad502/volumes" Aug 04 11:19:09.100417 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:09.100322 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99389820-45f5-43d7-92f8-1896bd1b6953" path="/var/lib/kubelet/pods/99389820-45f5-43d7-92f8-1896bd1b6953/volumes" Aug 04 11:19:15.156494 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:15.156446 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8" Aug 04 11:19:15.156958 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:15.156530 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8" Aug 04 11:19:15.169584 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:15.169546 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8" Aug 04 11:19:15.513489 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:15.513398 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8" Aug 04 11:19:25.937934 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:25.937893 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb"] Aug 04 11:19:25.938340 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:25.938285 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17cc3608-2b1a-4382-b717-5a9aabaad502" containerName="main" Aug 04 11:19:25.938340 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:25.938298 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="17cc3608-2b1a-4382-b717-5a9aabaad502" containerName="main" Aug 04 11:19:25.938340 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:25.938315 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99389820-45f5-43d7-92f8-1896bd1b6953" containerName="main" Aug 04 11:19:25.938340 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:25.938320 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="99389820-45f5-43d7-92f8-1896bd1b6953" containerName="main" Aug 04 11:19:25.938508 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:25.938375 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="17cc3608-2b1a-4382-b717-5a9aabaad502" containerName="main" Aug 04 11:19:25.938508 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:25.938385 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="99389820-45f5-43d7-92f8-1896bd1b6953" containerName="main" Aug 04 11:19:25.946859 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:25.946834 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb" Aug 04 11:19:25.950259 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:25.950229 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb"] Aug 04 11:19:25.951371 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:25.951338 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-460cac\"/\"kube-root-ca.crt\"" Aug 04 11:19:25.951507 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:25.951367 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-460cac\"/\"rate-test-model-460cac-kserve-self-signed-certs\"" Aug 04 11:19:25.951507 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:25.951416 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-460cac\"/\"openshift-service-ca.crt\"" Aug 04 11:19:25.951659 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:25.951350 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-460cac\"/\"default-dockercfg-c7lzs\"" Aug 04 11:19:26.058212 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:26.058181 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9be32fc9-36fe-454c-b346-a43acb8a104e-home\") pod \"rate-test-model-460cac-kserve-599fcb9b47-slxsb\" (UID: \"9be32fc9-36fe-454c-b346-a43acb8a104e\") " pod="ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb" Aug 04 11:19:26.058409 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:26.058283 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9be32fc9-36fe-454c-b346-a43acb8a104e-tls-certs\") pod \"rate-test-model-460cac-kserve-599fcb9b47-slxsb\" (UID: \"9be32fc9-36fe-454c-b346-a43acb8a104e\") " pod="ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb" Aug 04 11:19:26.058409 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:26.058303 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be32fc9-36fe-454c-b346-a43acb8a104e-dshm\") pod \"rate-test-model-460cac-kserve-599fcb9b47-slxsb\" (UID: \"9be32fc9-36fe-454c-b346-a43acb8a104e\") " pod="ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb" Aug 04 11:19:26.058409 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:26.058325 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be32fc9-36fe-454c-b346-a43acb8a104e-model-cache\") pod \"rate-test-model-460cac-kserve-599fcb9b47-slxsb\" (UID: \"9be32fc9-36fe-454c-b346-a43acb8a104e\") " pod="ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb" Aug 04 11:19:26.058409 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:26.058348 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be32fc9-36fe-454c-b346-a43acb8a104e-tmp-dir\") pod \"rate-test-model-460cac-kserve-599fcb9b47-slxsb\" (UID: \"9be32fc9-36fe-454c-b346-a43acb8a104e\") " pod="ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb" Aug 04 11:19:26.058409 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:26.058371 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xns6q\" (UniqueName: \"kubernetes.io/projected/9be32fc9-36fe-454c-b346-a43acb8a104e-kube-api-access-xns6q\") pod \"rate-test-model-460cac-kserve-599fcb9b47-slxsb\" (UID: \"9be32fc9-36fe-454c-b346-a43acb8a104e\") " pod="ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb" Aug 04 11:19:26.159618 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:26.159540 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9be32fc9-36fe-454c-b346-a43acb8a104e-tls-certs\") pod \"rate-test-model-460cac-kserve-599fcb9b47-slxsb\" (UID: \"9be32fc9-36fe-454c-b346-a43acb8a104e\") " pod="ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb" Aug 04 11:19:26.159830 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:26.159633 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be32fc9-36fe-454c-b346-a43acb8a104e-dshm\") pod \"rate-test-model-460cac-kserve-599fcb9b47-slxsb\" (UID: \"9be32fc9-36fe-454c-b346-a43acb8a104e\") " pod="ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb" Aug 04 11:19:26.159830 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:26.159658 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be32fc9-36fe-454c-b346-a43acb8a104e-model-cache\") pod \"rate-test-model-460cac-kserve-599fcb9b47-slxsb\" (UID: \"9be32fc9-36fe-454c-b346-a43acb8a104e\") " pod="ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb" Aug 04 11:19:26.159830 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:26.159673 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be32fc9-36fe-454c-b346-a43acb8a104e-tmp-dir\") pod \"rate-test-model-460cac-kserve-599fcb9b47-slxsb\" (UID: \"9be32fc9-36fe-454c-b346-a43acb8a104e\") " pod="ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb" Aug 04 11:19:26.159830 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:26.159695 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xns6q\" (UniqueName: \"kubernetes.io/projected/9be32fc9-36fe-454c-b346-a43acb8a104e-kube-api-access-xns6q\") pod \"rate-test-model-460cac-kserve-599fcb9b47-slxsb\" (UID: \"9be32fc9-36fe-454c-b346-a43acb8a104e\") " pod="ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb" Aug 04 11:19:26.159830 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:26.159757 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9be32fc9-36fe-454c-b346-a43acb8a104e-home\") pod \"rate-test-model-460cac-kserve-599fcb9b47-slxsb\" (UID: \"9be32fc9-36fe-454c-b346-a43acb8a104e\") " pod="ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb" Aug 04 11:19:26.160144 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:26.160122 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be32fc9-36fe-454c-b346-a43acb8a104e-model-cache\") pod \"rate-test-model-460cac-kserve-599fcb9b47-slxsb\" (UID: \"9be32fc9-36fe-454c-b346-a43acb8a104e\") " pod="ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb" Aug 04 11:19:26.160220 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:26.160175 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9be32fc9-36fe-454c-b346-a43acb8a104e-home\") pod \"rate-test-model-460cac-kserve-599fcb9b47-slxsb\" (UID: \"9be32fc9-36fe-454c-b346-a43acb8a104e\") " pod="ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb" Aug 04 11:19:26.160220 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:26.160206 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be32fc9-36fe-454c-b346-a43acb8a104e-tmp-dir\") pod \"rate-test-model-460cac-kserve-599fcb9b47-slxsb\" (UID: \"9be32fc9-36fe-454c-b346-a43acb8a104e\") " pod="ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb" Aug 04 11:19:26.162223 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:26.162199 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be32fc9-36fe-454c-b346-a43acb8a104e-dshm\") pod \"rate-test-model-460cac-kserve-599fcb9b47-slxsb\" (UID: \"9be32fc9-36fe-454c-b346-a43acb8a104e\") " pod="ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb" Aug 04 11:19:26.162341 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:26.162323 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9be32fc9-36fe-454c-b346-a43acb8a104e-tls-certs\") pod \"rate-test-model-460cac-kserve-599fcb9b47-slxsb\" (UID: \"9be32fc9-36fe-454c-b346-a43acb8a104e\") " pod="ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb" Aug 04 11:19:26.168193 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:26.168169 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xns6q\" (UniqueName: \"kubernetes.io/projected/9be32fc9-36fe-454c-b346-a43acb8a104e-kube-api-access-xns6q\") pod \"rate-test-model-460cac-kserve-599fcb9b47-slxsb\" (UID: \"9be32fc9-36fe-454c-b346-a43acb8a104e\") " pod="ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb" Aug 04 11:19:26.257811 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:26.257717 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb" Aug 04 11:19:26.400137 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:26.400111 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb"] Aug 04 11:19:26.402240 ip-10-0-142-113 kubenswrapper[2580]: W0804 11:19:26.402214 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9be32fc9_36fe_454c_b346_a43acb8a104e.slice/crio-b3616a4f9e3974663ce3e95f0802c15084fed2f1d34ba9b084b3066726e933b1 WatchSource:0}: Error finding container b3616a4f9e3974663ce3e95f0802c15084fed2f1d34ba9b084b3066726e933b1: Status 404 returned error can't find the container with id b3616a4f9e3974663ce3e95f0802c15084fed2f1d34ba9b084b3066726e933b1 Aug 04 11:19:26.557229 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:26.557132 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb" event={"ID":"9be32fc9-36fe-454c-b346-a43acb8a104e","Type":"ContainerStarted","Data":"ae66ae71b331c71a4cd5061af0d48adf53c84dd7b03b17ea83cc93307448fb01"} Aug 04 11:19:26.557229 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:26.557179 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb" event={"ID":"9be32fc9-36fe-454c-b346-a43acb8a104e","Type":"ContainerStarted","Data":"b3616a4f9e3974663ce3e95f0802c15084fed2f1d34ba9b084b3066726e933b1"} Aug 04 11:19:26.577124 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:26.577044 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb" podStartSLOduration=1.577028523 podStartE2EDuration="1.577028523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:19:26.576013608 +0000 UTC m=+3105.996901019" watchObservedRunningTime="2026-08-04 11:19:26.577028523 +0000 UTC m=+3105.997915934" Aug 04 11:19:28.526871 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:28.526837 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:19:36.258158 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:36.258120 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb" Aug 04 11:19:36.258678 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:36.258172 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb" Aug 04 11:19:36.270406 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:36.270377 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb" Aug 04 11:19:36.618960 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:19:36.618927 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb" Aug 04 11:20:05.349769 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:20:05.349671 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:20:13.648846 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:20:13.648806 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:20:40.159078 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:20:40.159040 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:20:45.606633 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:20:45.606595 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8"] Aug 04 11:20:46.552578 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:20:46.552526 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb"] Aug 04 11:20:49.272008 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:20:49.271977 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:21:00.626033 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:00.625959 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8" podUID="b5a82d0c-de1c-4011-8d2d-8fd665006701" containerName="main" containerID="cri-o://1ac11aac39eb48fe9871b9a5450e867147f5df0c88296e40a437e88ab14850d1" gracePeriod=45 Aug 04 11:21:00.880463 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:00.880396 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8" Aug 04 11:21:00.950148 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:00.950108 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5a82d0c-de1c-4011-8d2d-8fd665006701-home\") pod \"b5a82d0c-de1c-4011-8d2d-8fd665006701\" (UID: \"b5a82d0c-de1c-4011-8d2d-8fd665006701\") " Aug 04 11:21:00.950148 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:00.950146 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5a82d0c-de1c-4011-8d2d-8fd665006701-dshm\") pod \"b5a82d0c-de1c-4011-8d2d-8fd665006701\" (UID: \"b5a82d0c-de1c-4011-8d2d-8fd665006701\") " Aug 04 11:21:00.950353 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:00.950178 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5a82d0c-de1c-4011-8d2d-8fd665006701-tls-certs\") pod \"b5a82d0c-de1c-4011-8d2d-8fd665006701\" (UID: \"b5a82d0c-de1c-4011-8d2d-8fd665006701\") " Aug 04 11:21:00.950353 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:00.950204 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5a82d0c-de1c-4011-8d2d-8fd665006701-model-cache\") pod \"b5a82d0c-de1c-4011-8d2d-8fd665006701\" (UID: \"b5a82d0c-de1c-4011-8d2d-8fd665006701\") " Aug 04 11:21:00.950421 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:00.950398 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a82d0c-de1c-4011-8d2d-8fd665006701-home" (OuterVolumeSpecName: "home") pod "b5a82d0c-de1c-4011-8d2d-8fd665006701" (UID: "b5a82d0c-de1c-4011-8d2d-8fd665006701"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:21:00.950579 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:00.950528 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a82d0c-de1c-4011-8d2d-8fd665006701-model-cache" (OuterVolumeSpecName: "model-cache") pod "b5a82d0c-de1c-4011-8d2d-8fd665006701" (UID: "b5a82d0c-de1c-4011-8d2d-8fd665006701"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:21:00.952421 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:00.952399 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a82d0c-de1c-4011-8d2d-8fd665006701-dshm" (OuterVolumeSpecName: "dshm") pod "b5a82d0c-de1c-4011-8d2d-8fd665006701" (UID: "b5a82d0c-de1c-4011-8d2d-8fd665006701"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:21:00.952508 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:00.952414 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5a82d0c-de1c-4011-8d2d-8fd665006701-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b5a82d0c-de1c-4011-8d2d-8fd665006701" (UID: "b5a82d0c-de1c-4011-8d2d-8fd665006701"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:21:01.014093 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.014057 2580 generic.go:356] "Generic (PLEG): container finished" podID="b5a82d0c-de1c-4011-8d2d-8fd665006701" containerID="1ac11aac39eb48fe9871b9a5450e867147f5df0c88296e40a437e88ab14850d1" exitCode=0 Aug 04 11:21:01.014272 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.014103 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8" Aug 04 11:21:01.014272 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.014144 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8" event={"ID":"b5a82d0c-de1c-4011-8d2d-8fd665006701","Type":"ContainerDied","Data":"1ac11aac39eb48fe9871b9a5450e867147f5df0c88296e40a437e88ab14850d1"} Aug 04 11:21:01.014272 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.014187 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8" event={"ID":"b5a82d0c-de1c-4011-8d2d-8fd665006701","Type":"ContainerDied","Data":"c345244af077d0a985eedd9c2b78480db329a3a1c51a73f1c0b8c7387dda14ae"} Aug 04 11:21:01.014272 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.014204 2580 scope.go:117] "RemoveContainer" containerID="1ac11aac39eb48fe9871b9a5450e867147f5df0c88296e40a437e88ab14850d1" Aug 04 11:21:01.022949 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.022931 2580 scope.go:117] "RemoveContainer" containerID="1ac11aac39eb48fe9871b9a5450e867147f5df0c88296e40a437e88ab14850d1" Aug 04 11:21:01.023223 ip-10-0-142-113 kubenswrapper[2580]: E0804 11:21:01.023204 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ac11aac39eb48fe9871b9a5450e867147f5df0c88296e40a437e88ab14850d1\": container with ID starting with 1ac11aac39eb48fe9871b9a5450e867147f5df0c88296e40a437e88ab14850d1 not found: ID does not exist" containerID="1ac11aac39eb48fe9871b9a5450e867147f5df0c88296e40a437e88ab14850d1" Aug 04 11:21:01.023276 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.023232 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ac11aac39eb48fe9871b9a5450e867147f5df0c88296e40a437e88ab14850d1"} err="failed to get container status \"1ac11aac39eb48fe9871b9a5450e867147f5df0c88296e40a437e88ab14850d1\": rpc error: code = NotFound desc = could not find container \"1ac11aac39eb48fe9871b9a5450e867147f5df0c88296e40a437e88ab14850d1\": container with ID starting with 1ac11aac39eb48fe9871b9a5450e867147f5df0c88296e40a437e88ab14850d1 not found: ID does not exist" Aug 04 11:21:01.051288 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.051262 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slswj\" (UniqueName: \"kubernetes.io/projected/b5a82d0c-de1c-4011-8d2d-8fd665006701-kube-api-access-slswj\") pod \"b5a82d0c-de1c-4011-8d2d-8fd665006701\" (UID: \"b5a82d0c-de1c-4011-8d2d-8fd665006701\") " Aug 04 11:21:01.051438 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.051298 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5a82d0c-de1c-4011-8d2d-8fd665006701-tmp-dir\") pod \"b5a82d0c-de1c-4011-8d2d-8fd665006701\" (UID: \"b5a82d0c-de1c-4011-8d2d-8fd665006701\") " Aug 04 11:21:01.051525 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.051514 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5a82d0c-de1c-4011-8d2d-8fd665006701-tls-certs\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:21:01.051576 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.051527 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5a82d0c-de1c-4011-8d2d-8fd665006701-model-cache\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:21:01.051576 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.051536 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5a82d0c-de1c-4011-8d2d-8fd665006701-home\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:21:01.051576 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.051542 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5a82d0c-de1c-4011-8d2d-8fd665006701-dshm\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:21:01.051694 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.051603 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a82d0c-de1c-4011-8d2d-8fd665006701-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b5a82d0c-de1c-4011-8d2d-8fd665006701" (UID: "b5a82d0c-de1c-4011-8d2d-8fd665006701"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:21:01.053499 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.053472 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a82d0c-de1c-4011-8d2d-8fd665006701-kube-api-access-slswj" (OuterVolumeSpecName: "kube-api-access-slswj") pod "b5a82d0c-de1c-4011-8d2d-8fd665006701" (UID: "b5a82d0c-de1c-4011-8d2d-8fd665006701"). InnerVolumeSpecName "kube-api-access-slswj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:21:01.152592 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.152464 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slswj\" (UniqueName: \"kubernetes.io/projected/b5a82d0c-de1c-4011-8d2d-8fd665006701-kube-api-access-slswj\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:21:01.152592 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.152496 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5a82d0c-de1c-4011-8d2d-8fd665006701-tmp-dir\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:21:01.331482 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.331445 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8"] Aug 04 11:21:01.333794 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.333765 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-bc4a78/rate-test-model-bc4a78-kserve-598c645b79-cq2d8"] Aug 04 11:21:01.572244 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.572149 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb" podUID="9be32fc9-36fe-454c-b346-a43acb8a104e" containerName="main" containerID="cri-o://ae66ae71b331c71a4cd5061af0d48adf53c84dd7b03b17ea83cc93307448fb01" gracePeriod=45 Aug 04 11:21:01.828849 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.828782 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb" Aug 04 11:21:01.858719 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.858682 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be32fc9-36fe-454c-b346-a43acb8a104e-model-cache\") pod \"9be32fc9-36fe-454c-b346-a43acb8a104e\" (UID: \"9be32fc9-36fe-454c-b346-a43acb8a104e\") " Aug 04 11:21:01.858913 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.858742 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xns6q\" (UniqueName: \"kubernetes.io/projected/9be32fc9-36fe-454c-b346-a43acb8a104e-kube-api-access-xns6q\") pod \"9be32fc9-36fe-454c-b346-a43acb8a104e\" (UID: \"9be32fc9-36fe-454c-b346-a43acb8a104e\") " Aug 04 11:21:01.858913 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.858771 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be32fc9-36fe-454c-b346-a43acb8a104e-tmp-dir\") pod \"9be32fc9-36fe-454c-b346-a43acb8a104e\" (UID: \"9be32fc9-36fe-454c-b346-a43acb8a104e\") " Aug 04 11:21:01.858913 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.858827 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9be32fc9-36fe-454c-b346-a43acb8a104e-home\") pod \"9be32fc9-36fe-454c-b346-a43acb8a104e\" (UID: \"9be32fc9-36fe-454c-b346-a43acb8a104e\") " Aug 04 11:21:01.859058 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.858989 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be32fc9-36fe-454c-b346-a43acb8a104e-model-cache" (OuterVolumeSpecName: "model-cache") pod "9be32fc9-36fe-454c-b346-a43acb8a104e" (UID: "9be32fc9-36fe-454c-b346-a43acb8a104e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:21:01.859108 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.859094 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be32fc9-36fe-454c-b346-a43acb8a104e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9be32fc9-36fe-454c-b346-a43acb8a104e" (UID: "9be32fc9-36fe-454c-b346-a43acb8a104e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:21:01.859172 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.859149 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be32fc9-36fe-454c-b346-a43acb8a104e-model-cache\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:21:01.859211 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.859186 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be32fc9-36fe-454c-b346-a43acb8a104e-home" (OuterVolumeSpecName: "home") pod "9be32fc9-36fe-454c-b346-a43acb8a104e" (UID: "9be32fc9-36fe-454c-b346-a43acb8a104e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:21:01.860993 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.860971 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9be32fc9-36fe-454c-b346-a43acb8a104e-kube-api-access-xns6q" (OuterVolumeSpecName: "kube-api-access-xns6q") pod "9be32fc9-36fe-454c-b346-a43acb8a104e" (UID: "9be32fc9-36fe-454c-b346-a43acb8a104e"). InnerVolumeSpecName "kube-api-access-xns6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:21:01.959501 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.959452 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be32fc9-36fe-454c-b346-a43acb8a104e-dshm\") pod \"9be32fc9-36fe-454c-b346-a43acb8a104e\" (UID: \"9be32fc9-36fe-454c-b346-a43acb8a104e\") " Aug 04 11:21:01.959501 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.959496 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9be32fc9-36fe-454c-b346-a43acb8a104e-tls-certs\") pod \"9be32fc9-36fe-454c-b346-a43acb8a104e\" (UID: \"9be32fc9-36fe-454c-b346-a43acb8a104e\") " Aug 04 11:21:01.959751 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.959715 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be32fc9-36fe-454c-b346-a43acb8a104e-tmp-dir\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:21:01.959751 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.959726 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9be32fc9-36fe-454c-b346-a43acb8a104e-home\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:21:01.959751 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.959735 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xns6q\" (UniqueName: \"kubernetes.io/projected/9be32fc9-36fe-454c-b346-a43acb8a104e-kube-api-access-xns6q\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:21:01.961844 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.961811 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be32fc9-36fe-454c-b346-a43acb8a104e-dshm" (OuterVolumeSpecName: "dshm") pod "9be32fc9-36fe-454c-b346-a43acb8a104e" (UID: "9be32fc9-36fe-454c-b346-a43acb8a104e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:21:01.961964 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:01.961881 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9be32fc9-36fe-454c-b346-a43acb8a104e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9be32fc9-36fe-454c-b346-a43acb8a104e" (UID: "9be32fc9-36fe-454c-b346-a43acb8a104e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:21:02.020287 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:02.020246 2580 generic.go:356] "Generic (PLEG): container finished" podID="9be32fc9-36fe-454c-b346-a43acb8a104e" containerID="ae66ae71b331c71a4cd5061af0d48adf53c84dd7b03b17ea83cc93307448fb01" exitCode=0 Aug 04 11:21:02.020477 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:02.020292 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb" Aug 04 11:21:02.020477 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:02.020332 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb" event={"ID":"9be32fc9-36fe-454c-b346-a43acb8a104e","Type":"ContainerDied","Data":"ae66ae71b331c71a4cd5061af0d48adf53c84dd7b03b17ea83cc93307448fb01"} Aug 04 11:21:02.020477 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:02.020369 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb" event={"ID":"9be32fc9-36fe-454c-b346-a43acb8a104e","Type":"ContainerDied","Data":"b3616a4f9e3974663ce3e95f0802c15084fed2f1d34ba9b084b3066726e933b1"} Aug 04 11:21:02.020477 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:02.020394 2580 scope.go:117] "RemoveContainer" containerID="ae66ae71b331c71a4cd5061af0d48adf53c84dd7b03b17ea83cc93307448fb01" Aug 04 11:21:02.030008 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:02.029990 2580 scope.go:117] "RemoveContainer" containerID="ae66ae71b331c71a4cd5061af0d48adf53c84dd7b03b17ea83cc93307448fb01" Aug 04 11:21:02.030332 ip-10-0-142-113 kubenswrapper[2580]: E0804 11:21:02.030306 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae66ae71b331c71a4cd5061af0d48adf53c84dd7b03b17ea83cc93307448fb01\": container with ID starting with ae66ae71b331c71a4cd5061af0d48adf53c84dd7b03b17ea83cc93307448fb01 not found: ID does not exist" containerID="ae66ae71b331c71a4cd5061af0d48adf53c84dd7b03b17ea83cc93307448fb01" Aug 04 11:21:02.030402 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:02.030342 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae66ae71b331c71a4cd5061af0d48adf53c84dd7b03b17ea83cc93307448fb01"} err="failed to get container status \"ae66ae71b331c71a4cd5061af0d48adf53c84dd7b03b17ea83cc93307448fb01\": rpc error: code = NotFound desc = could not find container \"ae66ae71b331c71a4cd5061af0d48adf53c84dd7b03b17ea83cc93307448fb01\": container with ID starting with ae66ae71b331c71a4cd5061af0d48adf53c84dd7b03b17ea83cc93307448fb01 not found: ID does not exist" Aug 04 11:21:02.042737 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:02.042704 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb"] Aug 04 11:21:02.046053 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:02.046023 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-460cac/rate-test-model-460cac-kserve-599fcb9b47-slxsb"] Aug 04 11:21:02.060613 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:02.060584 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be32fc9-36fe-454c-b346-a43acb8a104e-dshm\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:21:02.060613 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:02.060614 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9be32fc9-36fe-454c-b346-a43acb8a104e-tls-certs\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:21:03.099433 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:03.099397 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9be32fc9-36fe-454c-b346-a43acb8a104e" path="/var/lib/kubelet/pods/9be32fc9-36fe-454c-b346-a43acb8a104e/volumes" Aug 04 11:21:03.099837 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:03.099816 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5a82d0c-de1c-4011-8d2d-8fd665006701" path="/var/lib/kubelet/pods/b5a82d0c-de1c-4011-8d2d-8fd665006701/volumes" Aug 04 11:21:06.226877 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.226841 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9"] Aug 04 11:21:06.227279 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.227203 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5a82d0c-de1c-4011-8d2d-8fd665006701" containerName="main" Aug 04 11:21:06.227279 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.227213 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5a82d0c-de1c-4011-8d2d-8fd665006701" containerName="main" Aug 04 11:21:06.227279 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.227221 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9be32fc9-36fe-454c-b346-a43acb8a104e" containerName="main" Aug 04 11:21:06.227279 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.227228 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9be32fc9-36fe-454c-b346-a43acb8a104e" containerName="main" Aug 04 11:21:06.227431 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.227286 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9be32fc9-36fe-454c-b346-a43acb8a104e" containerName="main" Aug 04 11:21:06.227431 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.227294 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5a82d0c-de1c-4011-8d2d-8fd665006701" containerName="main" Aug 04 11:21:06.231638 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.231615 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:06.235078 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.235058 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-1cb57d-openshift-default-dockercfg-5f44v\"" Aug 04 11:21:06.249663 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.249634 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9"] Aug 04 11:21:06.295965 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.295930 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d44af1c-5776-4a5e-8f99-c9538db0040b-credential-socket\") pod \"e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:06.295965 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.295967 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr826\" (UniqueName: \"kubernetes.io/projected/7d44af1c-5776-4a5e-8f99-c9538db0040b-kube-api-access-gr826\") pod \"e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:06.296187 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.295995 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d44af1c-5776-4a5e-8f99-c9538db0040b-workload-certs\") pod \"e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:06.296187 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.296069 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d44af1c-5776-4a5e-8f99-c9538db0040b-workload-socket\") pod \"e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:06.296187 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.296107 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d44af1c-5776-4a5e-8f99-c9538db0040b-istio-podinfo\") pod \"e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:06.296187 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.296144 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d44af1c-5776-4a5e-8f99-c9538db0040b-istiod-ca-cert\") pod \"e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:06.296187 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.296158 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d44af1c-5776-4a5e-8f99-c9538db0040b-istio-envoy\") pod \"e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:06.296187 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.296179 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d44af1c-5776-4a5e-8f99-c9538db0040b-istio-token\") pod \"e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:06.296380 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.296264 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d44af1c-5776-4a5e-8f99-c9538db0040b-istio-data\") pod \"e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:06.396923 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.396878 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d44af1c-5776-4a5e-8f99-c9538db0040b-credential-socket\") pod \"e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:06.396923 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.396922 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr826\" (UniqueName: \"kubernetes.io/projected/7d44af1c-5776-4a5e-8f99-c9538db0040b-kube-api-access-gr826\") pod \"e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:06.397149 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.396966 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d44af1c-5776-4a5e-8f99-c9538db0040b-workload-certs\") pod \"e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:06.397149 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.397004 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d44af1c-5776-4a5e-8f99-c9538db0040b-workload-socket\") pod \"e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:06.397149 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.397031 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d44af1c-5776-4a5e-8f99-c9538db0040b-istio-podinfo\") pod \"e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:06.397282 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.397182 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d44af1c-5776-4a5e-8f99-c9538db0040b-istiod-ca-cert\") pod \"e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:06.397282 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.397231 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d44af1c-5776-4a5e-8f99-c9538db0040b-istio-envoy\") pod \"e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:06.397384 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.397287 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d44af1c-5776-4a5e-8f99-c9538db0040b-istio-token\") pod \"e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:06.397384 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.397370 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d44af1c-5776-4a5e-8f99-c9538db0040b-credential-socket\") pod \"e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:06.397495 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.397386 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d44af1c-5776-4a5e-8f99-c9538db0040b-istio-data\") pod \"e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:06.397495 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.397431 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d44af1c-5776-4a5e-8f99-c9538db0040b-workload-certs\") pod \"e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:06.397495 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.397484 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d44af1c-5776-4a5e-8f99-c9538db0040b-workload-socket\") pod \"e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:06.397731 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.397712 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d44af1c-5776-4a5e-8f99-c9538db0040b-istio-data\") pod \"e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:06.397922 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.397903 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d44af1c-5776-4a5e-8f99-c9538db0040b-istiod-ca-cert\") pod \"e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:06.399652 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.399629 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d44af1c-5776-4a5e-8f99-c9538db0040b-istio-envoy\") pod \"e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:06.399748 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.399704 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d44af1c-5776-4a5e-8f99-c9538db0040b-istio-podinfo\") pod \"e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:06.406955 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.406928 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d44af1c-5776-4a5e-8f99-c9538db0040b-istio-token\") pod \"e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:06.408039 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.408020 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr826\" (UniqueName: \"kubernetes.io/projected/7d44af1c-5776-4a5e-8f99-c9538db0040b-kube-api-access-gr826\") pod \"e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:06.542339 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.542246 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:06.691425 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.691399 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9"] Aug 04 11:21:06.694601 ip-10-0-142-113 kubenswrapper[2580]: W0804 11:21:06.694572 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d44af1c_5776_4a5e_8f99_c9538db0040b.slice/crio-ea66e2a14910c6b300de830401e5c5bf9e36c9eb5f0ddbfa2eed2d179ec932f1 WatchSource:0}: Error finding container ea66e2a14910c6b300de830401e5c5bf9e36c9eb5f0ddbfa2eed2d179ec932f1: Status 404 returned error can't find the container with id ea66e2a14910c6b300de830401e5c5bf9e36c9eb5f0ddbfa2eed2d179ec932f1 Aug 04 11:21:06.696871 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.696838 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:21:06.696927 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.696904 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:21:06.696959 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:06.696937 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:21:07.049834 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:07.049804 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" event={"ID":"7d44af1c-5776-4a5e-8f99-c9538db0040b","Type":"ContainerStarted","Data":"13b11b983e2c653c028a8d7aba420ab8d936b259656d378ebd4926f4cac5a436"} Aug 04 11:21:07.050018 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:07.049852 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" event={"ID":"7d44af1c-5776-4a5e-8f99-c9538db0040b","Type":"ContainerStarted","Data":"ea66e2a14910c6b300de830401e5c5bf9e36c9eb5f0ddbfa2eed2d179ec932f1"} Aug 04 11:21:07.073769 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:07.073717 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" podStartSLOduration=1.073701287 podStartE2EDuration="1.073701287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:21:07.072537442 +0000 UTC m=+3206.493424852" watchObservedRunningTime="2026-08-04 11:21:07.073701287 +0000 UTC m=+3206.494588676" Aug 04 11:21:07.543399 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:07.543354 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:07.548412 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:07.548381 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:08.054452 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:08.054421 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:08.055595 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:08.055574 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:13.835899 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:13.835862 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:21:23.765024 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:23.764312 2580 patch_prober.go:28] interesting pod/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": read tcp 10.133.0.2:49224->10.133.0.62:15021: read: connection reset by peer" start-of-body= Aug 04 11:21:23.765024 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:23.764393 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" podUID="7d44af1c-5776-4a5e-8f99-c9538db0040b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": read tcp 10.133.0.2:49224->10.133.0.62:15021: read: connection reset by peer" Aug 04 11:21:24.141384 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:24.141350 2580 generic.go:356] "Generic (PLEG): container finished" podID="7d44af1c-5776-4a5e-8f99-c9538db0040b" containerID="13b11b983e2c653c028a8d7aba420ab8d936b259656d378ebd4926f4cac5a436" exitCode=0 Aug 04 11:21:24.141597 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:24.141439 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" event={"ID":"7d44af1c-5776-4a5e-8f99-c9538db0040b","Type":"ContainerDied","Data":"13b11b983e2c653c028a8d7aba420ab8d936b259656d378ebd4926f4cac5a436"} Aug 04 11:21:24.141888 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:24.141873 2580 scope.go:117] "RemoveContainer" containerID="13b11b983e2c653c028a8d7aba420ab8d936b259656d378ebd4926f4cac5a436" Aug 04 11:21:24.143103 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:24.143069 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:21:24.143195 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:24.143138 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:21:24.143195 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:24.143186 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:21:25.147625 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:25.147593 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" event={"ID":"7d44af1c-5776-4a5e-8f99-c9538db0040b","Type":"ContainerStarted","Data":"8e4ff3463b17b6ada6721e822bf82defb30bf4750dd3f5e14b731c52a3935315"} Aug 04 11:21:25.542876 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:25.542782 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:25.543066 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:25.543042 2580 patch_prober.go:28] interesting pod/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Aug 04 11:21:25.543139 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:25.543095 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" podUID="7d44af1c-5776-4a5e-8f99-c9538db0040b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Aug 04 11:21:26.543652 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:26.543611 2580 patch_prober.go:28] interesting pod/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Aug 04 11:21:26.544126 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:26.543678 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" podUID="7d44af1c-5776-4a5e-8f99-c9538db0040b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Aug 04 11:21:27.542954 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:27.542907 2580 patch_prober.go:28] interesting pod/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Aug 04 11:21:27.543176 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:27.542976 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" podUID="7d44af1c-5776-4a5e-8f99-c9538db0040b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Aug 04 11:21:28.542921 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:28.542882 2580 patch_prober.go:28] interesting pod/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Aug 04 11:21:28.543318 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:28.542942 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" podUID="7d44af1c-5776-4a5e-8f99-c9538db0040b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Aug 04 11:21:29.542642 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:29.542601 2580 patch_prober.go:28] interesting pod/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Aug 04 11:21:29.542832 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:29.542660 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" podUID="7d44af1c-5776-4a5e-8f99-c9538db0040b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Aug 04 11:21:30.547169 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:30.547135 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:30.547547 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:30.547436 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:30.548088 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:30.548072 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:21:35.046114 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.046075 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj"] Aug 04 11:21:35.049634 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.049610 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:35.052587 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.052538 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-651a7a-openshift-default-dockercfg-gnkgk\"" Aug 04 11:21:35.061267 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.061237 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj"] Aug 04 11:21:35.184815 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.184767 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db0acfe-a199-45d7-8f35-35235cc08cd3-credential-socket\") pod \"e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:35.185012 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.184831 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db0acfe-a199-45d7-8f35-35235cc08cd3-istiod-ca-cert\") pod \"e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:35.185012 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.184864 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db0acfe-a199-45d7-8f35-35235cc08cd3-istio-envoy\") pod \"e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:35.185012 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.184915 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db0acfe-a199-45d7-8f35-35235cc08cd3-istio-token\") pod \"e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:35.185012 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.184959 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db0acfe-a199-45d7-8f35-35235cc08cd3-workload-socket\") pod \"e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:35.185012 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.184985 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db0acfe-a199-45d7-8f35-35235cc08cd3-istio-podinfo\") pod \"e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:35.185305 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.185049 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db0acfe-a199-45d7-8f35-35235cc08cd3-istio-data\") pod \"e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:35.185305 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.185081 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm6x7\" (UniqueName: \"kubernetes.io/projected/6db0acfe-a199-45d7-8f35-35235cc08cd3-kube-api-access-gm6x7\") pod \"e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:35.185305 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.185109 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db0acfe-a199-45d7-8f35-35235cc08cd3-workload-certs\") pod \"e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:35.286142 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.286105 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db0acfe-a199-45d7-8f35-35235cc08cd3-istio-data\") pod \"e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:35.286142 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.286148 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm6x7\" (UniqueName: \"kubernetes.io/projected/6db0acfe-a199-45d7-8f35-35235cc08cd3-kube-api-access-gm6x7\") pod \"e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:35.286402 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.286171 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db0acfe-a199-45d7-8f35-35235cc08cd3-workload-certs\") pod \"e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:35.286402 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.286203 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db0acfe-a199-45d7-8f35-35235cc08cd3-credential-socket\") pod \"e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:35.286402 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.286227 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db0acfe-a199-45d7-8f35-35235cc08cd3-istiod-ca-cert\") pod \"e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:35.286402 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.286246 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db0acfe-a199-45d7-8f35-35235cc08cd3-istio-envoy\") pod \"e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:35.286402 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.286297 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db0acfe-a199-45d7-8f35-35235cc08cd3-istio-token\") pod \"e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:35.286402 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.286331 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db0acfe-a199-45d7-8f35-35235cc08cd3-workload-socket\") pod \"e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:35.286402 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.286347 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db0acfe-a199-45d7-8f35-35235cc08cd3-istio-podinfo\") pod \"e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:35.286811 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.286625 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db0acfe-a199-45d7-8f35-35235cc08cd3-workload-certs\") pod \"e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:35.286870 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.286813 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db0acfe-a199-45d7-8f35-35235cc08cd3-credential-socket\") pod \"e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:35.286870 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.286835 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db0acfe-a199-45d7-8f35-35235cc08cd3-istio-data\") pod \"e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:35.287102 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.287078 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db0acfe-a199-45d7-8f35-35235cc08cd3-istiod-ca-cert\") pod \"e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:35.287198 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.287151 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db0acfe-a199-45d7-8f35-35235cc08cd3-workload-socket\") pod \"e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:35.289035 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.289013 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db0acfe-a199-45d7-8f35-35235cc08cd3-istio-envoy\") pod \"e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:35.289146 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.289064 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db0acfe-a199-45d7-8f35-35235cc08cd3-istio-podinfo\") pod \"e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:35.295128 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.295104 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db0acfe-a199-45d7-8f35-35235cc08cd3-istio-token\") pod \"e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:35.295304 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.295283 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm6x7\" (UniqueName: \"kubernetes.io/projected/6db0acfe-a199-45d7-8f35-35235cc08cd3-kube-api-access-gm6x7\") pod \"e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:35.360122 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.360090 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:35.501128 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.501100 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj"] Aug 04 11:21:35.503721 ip-10-0-142-113 kubenswrapper[2580]: W0804 11:21:35.503691 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db0acfe_a199_45d7_8f35_35235cc08cd3.slice/crio-b201051a4b1fd846c291386fe94386d870789f7aac36b18e7d381c6211affdd8 WatchSource:0}: Error finding container b201051a4b1fd846c291386fe94386d870789f7aac36b18e7d381c6211affdd8: Status 404 returned error can't find the container with id b201051a4b1fd846c291386fe94386d870789f7aac36b18e7d381c6211affdd8 Aug 04 11:21:35.506282 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.506246 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:21:35.506367 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.506324 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:21:35.506367 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:35.506355 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:21:36.210769 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:36.210733 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" event={"ID":"6db0acfe-a199-45d7-8f35-35235cc08cd3","Type":"ContainerStarted","Data":"bf20c6b4fdfba70ed5636f158a3bbadd92d876badfe1375a7fb94eea2a27bb75"} Aug 04 11:21:36.211157 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:36.210786 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" event={"ID":"6db0acfe-a199-45d7-8f35-35235cc08cd3","Type":"ContainerStarted","Data":"b201051a4b1fd846c291386fe94386d870789f7aac36b18e7d381c6211affdd8"} Aug 04 11:21:36.238092 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:36.238041 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" podStartSLOduration=1.238023925 podStartE2EDuration="1.238023925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:21:36.237476322 +0000 UTC m=+3235.658363734" watchObservedRunningTime="2026-08-04 11:21:36.238023925 +0000 UTC m=+3235.658911399" Aug 04 11:21:36.361020 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:36.360875 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:36.365933 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:36.365906 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:37.216216 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:37.216186 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:37.217251 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:37.217233 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:42.733537 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:42.733499 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:21:52.615482 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:52.615419 2580 patch_prober.go:28] interesting pod/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": read tcp 10.133.0.2:58452->10.133.0.63:15021: read: connection reset by peer" start-of-body= Aug 04 11:21:52.615959 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:52.615491 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" podUID="6db0acfe-a199-45d7-8f35-35235cc08cd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": read tcp 10.133.0.2:58452->10.133.0.63:15021: read: connection reset by peer" Aug 04 11:21:53.305653 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:53.305619 2580 generic.go:356] "Generic (PLEG): container finished" podID="6db0acfe-a199-45d7-8f35-35235cc08cd3" containerID="bf20c6b4fdfba70ed5636f158a3bbadd92d876badfe1375a7fb94eea2a27bb75" exitCode=0 Aug 04 11:21:53.305823 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:53.305663 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" event={"ID":"6db0acfe-a199-45d7-8f35-35235cc08cd3","Type":"ContainerDied","Data":"bf20c6b4fdfba70ed5636f158a3bbadd92d876badfe1375a7fb94eea2a27bb75"} Aug 04 11:21:53.306057 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:53.306030 2580 scope.go:117] "RemoveContainer" containerID="bf20c6b4fdfba70ed5636f158a3bbadd92d876badfe1375a7fb94eea2a27bb75" Aug 04 11:21:53.307184 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:53.307157 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:21:53.307256 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:53.307243 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:21:53.307299 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:53.307286 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 04 11:21:54.313307 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:54.313273 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" event={"ID":"6db0acfe-a199-45d7-8f35-35235cc08cd3","Type":"ContainerStarted","Data":"4409ad6f3684c3844982eea401ba49638f602cc5948fee172dfff34eb64f320c"} Aug 04 11:21:54.361098 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:54.361061 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:21:54.361281 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:54.361258 2580 patch_prober.go:28] interesting pod/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Aug 04 11:21:54.361327 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:54.361308 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" podUID="6db0acfe-a199-45d7-8f35-35235cc08cd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Aug 04 11:21:55.361104 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:55.361066 2580 patch_prober.go:28] interesting pod/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Aug 04 11:21:55.361518 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:55.361124 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" podUID="6db0acfe-a199-45d7-8f35-35235cc08cd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Aug 04 11:21:56.361272 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:56.361233 2580 patch_prober.go:28] interesting pod/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Aug 04 11:21:56.361707 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:56.361288 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" podUID="6db0acfe-a199-45d7-8f35-35235cc08cd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Aug 04 11:21:57.361157 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:57.361111 2580 patch_prober.go:28] interesting pod/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Aug 04 11:21:57.361366 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:57.361176 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" podUID="6db0acfe-a199-45d7-8f35-35235cc08cd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Aug 04 11:21:58.360807 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:58.360764 2580 patch_prober.go:28] interesting pod/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Aug 04 11:21:58.360990 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:58.360820 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" podUID="6db0acfe-a199-45d7-8f35-35235cc08cd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Aug 04 11:21:59.360896 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:59.360849 2580 patch_prober.go:28] interesting pod/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Aug 04 11:21:59.361379 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:21:59.360909 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" podUID="6db0acfe-a199-45d7-8f35-35235cc08cd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Aug 04 11:22:00.364532 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:00.364502 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:22:00.364955 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:00.364812 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:22:00.365614 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:00.365597 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:22:17.533347 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:17.533309 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp"] Aug 04 11:22:17.536620 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:17.536597 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp" Aug 04 11:22:17.539536 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:17.539510 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-1cb57d\"/\"kube-root-ca.crt\"" Aug 04 11:22:17.540916 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:17.540893 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-1cb57d\"/\"ipp-route-1cb57d-kserve-self-signed-certs\"" Aug 04 11:22:17.541033 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:17.541008 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-1cb57d\"/\"openshift-service-ca.crt\"" Aug 04 11:22:17.541091 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:17.541057 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-1cb57d\"/\"default-dockercfg-m65tz\"" Aug 04 11:22:17.545223 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:17.545199 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp"] Aug 04 11:22:17.611134 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:17.611096 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a920567-45c8-43aa-a3a5-9335fcd3b007-home\") pod \"ipp-route-1cb57d-kserve-7755579ccf-zhjcp\" (UID: \"6a920567-45c8-43aa-a3a5-9335fcd3b007\") " pod="ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp" Aug 04 11:22:17.611134 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:17.611136 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjf5j\" (UniqueName: \"kubernetes.io/projected/6a920567-45c8-43aa-a3a5-9335fcd3b007-kube-api-access-mjf5j\") pod \"ipp-route-1cb57d-kserve-7755579ccf-zhjcp\" (UID: \"6a920567-45c8-43aa-a3a5-9335fcd3b007\") " pod="ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp" Aug 04 11:22:17.611408 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:17.611158 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a920567-45c8-43aa-a3a5-9335fcd3b007-dshm\") pod \"ipp-route-1cb57d-kserve-7755579ccf-zhjcp\" (UID: \"6a920567-45c8-43aa-a3a5-9335fcd3b007\") " pod="ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp" Aug 04 11:22:17.611408 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:17.611215 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a920567-45c8-43aa-a3a5-9335fcd3b007-model-cache\") pod \"ipp-route-1cb57d-kserve-7755579ccf-zhjcp\" (UID: \"6a920567-45c8-43aa-a3a5-9335fcd3b007\") " pod="ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp" Aug 04 11:22:17.611408 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:17.611256 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a920567-45c8-43aa-a3a5-9335fcd3b007-tls-certs\") pod \"ipp-route-1cb57d-kserve-7755579ccf-zhjcp\" (UID: \"6a920567-45c8-43aa-a3a5-9335fcd3b007\") " pod="ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp" Aug 04 11:22:17.611408 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:17.611398 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a920567-45c8-43aa-a3a5-9335fcd3b007-tmp-dir\") pod \"ipp-route-1cb57d-kserve-7755579ccf-zhjcp\" (UID: \"6a920567-45c8-43aa-a3a5-9335fcd3b007\") " pod="ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp" Aug 04 11:22:17.712849 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:17.712804 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a920567-45c8-43aa-a3a5-9335fcd3b007-tmp-dir\") pod \"ipp-route-1cb57d-kserve-7755579ccf-zhjcp\" (UID: \"6a920567-45c8-43aa-a3a5-9335fcd3b007\") " pod="ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp" Aug 04 11:22:17.713049 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:17.712862 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a920567-45c8-43aa-a3a5-9335fcd3b007-home\") pod \"ipp-route-1cb57d-kserve-7755579ccf-zhjcp\" (UID: \"6a920567-45c8-43aa-a3a5-9335fcd3b007\") " pod="ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp" Aug 04 11:22:17.713049 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:17.713030 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjf5j\" (UniqueName: \"kubernetes.io/projected/6a920567-45c8-43aa-a3a5-9335fcd3b007-kube-api-access-mjf5j\") pod \"ipp-route-1cb57d-kserve-7755579ccf-zhjcp\" (UID: \"6a920567-45c8-43aa-a3a5-9335fcd3b007\") " pod="ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp" Aug 04 11:22:17.713165 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:17.713078 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a920567-45c8-43aa-a3a5-9335fcd3b007-dshm\") pod \"ipp-route-1cb57d-kserve-7755579ccf-zhjcp\" (UID: \"6a920567-45c8-43aa-a3a5-9335fcd3b007\") " pod="ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp" Aug 04 11:22:17.713165 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:17.713116 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a920567-45c8-43aa-a3a5-9335fcd3b007-model-cache\") pod \"ipp-route-1cb57d-kserve-7755579ccf-zhjcp\" (UID: \"6a920567-45c8-43aa-a3a5-9335fcd3b007\") " pod="ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp" Aug 04 11:22:17.713165 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:17.713139 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a920567-45c8-43aa-a3a5-9335fcd3b007-tls-certs\") pod \"ipp-route-1cb57d-kserve-7755579ccf-zhjcp\" (UID: \"6a920567-45c8-43aa-a3a5-9335fcd3b007\") " pod="ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp" Aug 04 11:22:17.713331 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:17.713257 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a920567-45c8-43aa-a3a5-9335fcd3b007-home\") pod \"ipp-route-1cb57d-kserve-7755579ccf-zhjcp\" (UID: \"6a920567-45c8-43aa-a3a5-9335fcd3b007\") " pod="ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp" Aug 04 11:22:17.713331 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:17.713314 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a920567-45c8-43aa-a3a5-9335fcd3b007-tmp-dir\") pod \"ipp-route-1cb57d-kserve-7755579ccf-zhjcp\" (UID: \"6a920567-45c8-43aa-a3a5-9335fcd3b007\") " pod="ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp" Aug 04 11:22:17.713555 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:17.713532 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a920567-45c8-43aa-a3a5-9335fcd3b007-model-cache\") pod \"ipp-route-1cb57d-kserve-7755579ccf-zhjcp\" (UID: \"6a920567-45c8-43aa-a3a5-9335fcd3b007\") " pod="ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp" Aug 04 11:22:17.715548 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:17.715528 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a920567-45c8-43aa-a3a5-9335fcd3b007-dshm\") pod \"ipp-route-1cb57d-kserve-7755579ccf-zhjcp\" (UID: \"6a920567-45c8-43aa-a3a5-9335fcd3b007\") " pod="ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp" Aug 04 11:22:17.715953 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:17.715932 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a920567-45c8-43aa-a3a5-9335fcd3b007-tls-certs\") pod \"ipp-route-1cb57d-kserve-7755579ccf-zhjcp\" (UID: \"6a920567-45c8-43aa-a3a5-9335fcd3b007\") " pod="ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp" Aug 04 11:22:17.723366 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:17.723336 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjf5j\" (UniqueName: \"kubernetes.io/projected/6a920567-45c8-43aa-a3a5-9335fcd3b007-kube-api-access-mjf5j\") pod \"ipp-route-1cb57d-kserve-7755579ccf-zhjcp\" (UID: \"6a920567-45c8-43aa-a3a5-9335fcd3b007\") " pod="ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp" Aug 04 11:22:17.848763 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:17.848655 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp" Aug 04 11:22:17.989800 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:17.989769 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp"] Aug 04 11:22:17.991035 ip-10-0-142-113 kubenswrapper[2580]: W0804 11:22:17.991005 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a920567_45c8_43aa_a3a5_9335fcd3b007.slice/crio-ab253ab9d735658945fb129aee22a11b69e759a9b4604f30225c668b41f188ca WatchSource:0}: Error finding container ab253ab9d735658945fb129aee22a11b69e759a9b4604f30225c668b41f188ca: Status 404 returned error can't find the container with id ab253ab9d735658945fb129aee22a11b69e759a9b4604f30225c668b41f188ca Aug 04 11:22:18.432253 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:18.432214 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp" event={"ID":"6a920567-45c8-43aa-a3a5-9335fcd3b007","Type":"ContainerStarted","Data":"d10cf0a0d1a422068d83845df5854bb0e882900e15f6a36912188d90f0b80a41"} Aug 04 11:22:18.432439 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:18.432260 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp" event={"ID":"6a920567-45c8-43aa-a3a5-9335fcd3b007","Type":"ContainerStarted","Data":"ab253ab9d735658945fb129aee22a11b69e759a9b4604f30225c668b41f188ca"} Aug 04 11:22:18.454702 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:18.454644 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp" podStartSLOduration=1.454628525 podStartE2EDuration="1.454628525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:22:18.453160906 +0000 UTC m=+3277.874048333" watchObservedRunningTime="2026-08-04 11:22:18.454628525 +0000 UTC m=+3277.875515936" Aug 04 11:22:20.238433 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:20.238385 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:22:27.848815 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:27.848767 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp" Aug 04 11:22:27.849425 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:27.848848 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp" Aug 04 11:22:27.861531 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:27.861498 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp" Aug 04 11:22:28.490655 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:28.490615 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp" Aug 04 11:22:41.281921 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:41.281880 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tb4zk_c5922147-cb42-4f62-9880-8deaa51a4b0f/ovn-acl-logging/0.log" Aug 04 11:22:41.286016 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:41.285991 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tb4zk_c5922147-cb42-4f62-9880-8deaa51a4b0f/ovn-acl-logging/0.log" Aug 04 11:22:44.036529 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:22:44.036490 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:23:06.431230 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:23:06.431145 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:23:35.131503 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:23:35.131462 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:23:56.329501 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:23:56.329460 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:24:03.965227 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:03.965194 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp"] Aug 04 11:24:05.528259 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:05.528225 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:24:18.991382 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:18.991315 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp" podUID="6a920567-45c8-43aa-a3a5-9335fcd3b007" containerName="main" containerID="cri-o://d10cf0a0d1a422068d83845df5854bb0e882900e15f6a36912188d90f0b80a41" gracePeriod=45 Aug 04 11:24:19.242169 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:19.242107 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp" Aug 04 11:24:19.352153 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:19.352112 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a920567-45c8-43aa-a3a5-9335fcd3b007-home\") pod \"6a920567-45c8-43aa-a3a5-9335fcd3b007\" (UID: \"6a920567-45c8-43aa-a3a5-9335fcd3b007\") " Aug 04 11:24:19.352347 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:19.352198 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjf5j\" (UniqueName: \"kubernetes.io/projected/6a920567-45c8-43aa-a3a5-9335fcd3b007-kube-api-access-mjf5j\") pod \"6a920567-45c8-43aa-a3a5-9335fcd3b007\" (UID: \"6a920567-45c8-43aa-a3a5-9335fcd3b007\") " Aug 04 11:24:19.352347 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:19.352233 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a920567-45c8-43aa-a3a5-9335fcd3b007-tls-certs\") pod \"6a920567-45c8-43aa-a3a5-9335fcd3b007\" (UID: \"6a920567-45c8-43aa-a3a5-9335fcd3b007\") " Aug 04 11:24:19.352347 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:19.352255 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a920567-45c8-43aa-a3a5-9335fcd3b007-dshm\") pod \"6a920567-45c8-43aa-a3a5-9335fcd3b007\" (UID: \"6a920567-45c8-43aa-a3a5-9335fcd3b007\") " Aug 04 11:24:19.352347 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:19.352321 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a920567-45c8-43aa-a3a5-9335fcd3b007-model-cache\") pod \"6a920567-45c8-43aa-a3a5-9335fcd3b007\" (UID: \"6a920567-45c8-43aa-a3a5-9335fcd3b007\") " Aug 04 11:24:19.352604 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:19.352350 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a920567-45c8-43aa-a3a5-9335fcd3b007-tmp-dir\") pod \"6a920567-45c8-43aa-a3a5-9335fcd3b007\" (UID: \"6a920567-45c8-43aa-a3a5-9335fcd3b007\") " Aug 04 11:24:19.352604 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:19.352448 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a920567-45c8-43aa-a3a5-9335fcd3b007-home" (OuterVolumeSpecName: "home") pod "6a920567-45c8-43aa-a3a5-9335fcd3b007" (UID: "6a920567-45c8-43aa-a3a5-9335fcd3b007"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:24:19.352707 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:19.352662 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a920567-45c8-43aa-a3a5-9335fcd3b007-model-cache" (OuterVolumeSpecName: "model-cache") pod "6a920567-45c8-43aa-a3a5-9335fcd3b007" (UID: "6a920567-45c8-43aa-a3a5-9335fcd3b007"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:24:19.352761 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:19.352708 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a920567-45c8-43aa-a3a5-9335fcd3b007-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6a920567-45c8-43aa-a3a5-9335fcd3b007" (UID: "6a920567-45c8-43aa-a3a5-9335fcd3b007"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:24:19.352820 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:19.352781 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a920567-45c8-43aa-a3a5-9335fcd3b007-home\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:24:19.352820 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:19.352802 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a920567-45c8-43aa-a3a5-9335fcd3b007-model-cache\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:24:19.354653 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:19.354612 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a920567-45c8-43aa-a3a5-9335fcd3b007-dshm" (OuterVolumeSpecName: "dshm") pod "6a920567-45c8-43aa-a3a5-9335fcd3b007" (UID: "6a920567-45c8-43aa-a3a5-9335fcd3b007"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:24:19.354780 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:19.354712 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a920567-45c8-43aa-a3a5-9335fcd3b007-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a920567-45c8-43aa-a3a5-9335fcd3b007" (UID: "6a920567-45c8-43aa-a3a5-9335fcd3b007"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:24:19.354780 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:19.354725 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a920567-45c8-43aa-a3a5-9335fcd3b007-kube-api-access-mjf5j" (OuterVolumeSpecName: "kube-api-access-mjf5j") pod "6a920567-45c8-43aa-a3a5-9335fcd3b007" (UID: "6a920567-45c8-43aa-a3a5-9335fcd3b007"). InnerVolumeSpecName "kube-api-access-mjf5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:24:19.453821 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:19.453791 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjf5j\" (UniqueName: \"kubernetes.io/projected/6a920567-45c8-43aa-a3a5-9335fcd3b007-kube-api-access-mjf5j\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:24:19.453821 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:19.453818 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a920567-45c8-43aa-a3a5-9335fcd3b007-tls-certs\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:24:19.453821 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:19.453826 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a920567-45c8-43aa-a3a5-9335fcd3b007-dshm\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:24:19.454095 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:19.453834 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a920567-45c8-43aa-a3a5-9335fcd3b007-tmp-dir\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:24:20.032825 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:20.032785 2580 generic.go:356] "Generic (PLEG): container finished" podID="6a920567-45c8-43aa-a3a5-9335fcd3b007" containerID="d10cf0a0d1a422068d83845df5854bb0e882900e15f6a36912188d90f0b80a41" exitCode=0 Aug 04 11:24:20.033240 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:20.032833 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp" Aug 04 11:24:20.033240 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:20.032871 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp" event={"ID":"6a920567-45c8-43aa-a3a5-9335fcd3b007","Type":"ContainerDied","Data":"d10cf0a0d1a422068d83845df5854bb0e882900e15f6a36912188d90f0b80a41"} Aug 04 11:24:20.033240 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:20.032917 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp" event={"ID":"6a920567-45c8-43aa-a3a5-9335fcd3b007","Type":"ContainerDied","Data":"ab253ab9d735658945fb129aee22a11b69e759a9b4604f30225c668b41f188ca"} Aug 04 11:24:20.033240 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:20.032936 2580 scope.go:117] "RemoveContainer" containerID="d10cf0a0d1a422068d83845df5854bb0e882900e15f6a36912188d90f0b80a41" Aug 04 11:24:20.041982 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:20.041965 2580 scope.go:117] "RemoveContainer" containerID="d10cf0a0d1a422068d83845df5854bb0e882900e15f6a36912188d90f0b80a41" Aug 04 11:24:20.042231 ip-10-0-142-113 kubenswrapper[2580]: E0804 11:24:20.042210 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d10cf0a0d1a422068d83845df5854bb0e882900e15f6a36912188d90f0b80a41\": container with ID starting with d10cf0a0d1a422068d83845df5854bb0e882900e15f6a36912188d90f0b80a41 not found: ID does not exist" containerID="d10cf0a0d1a422068d83845df5854bb0e882900e15f6a36912188d90f0b80a41" Aug 04 11:24:20.042270 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:20.042241 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d10cf0a0d1a422068d83845df5854bb0e882900e15f6a36912188d90f0b80a41"} err="failed to get container status \"d10cf0a0d1a422068d83845df5854bb0e882900e15f6a36912188d90f0b80a41\": rpc error: code = NotFound desc = could not find container \"d10cf0a0d1a422068d83845df5854bb0e882900e15f6a36912188d90f0b80a41\": container with ID starting with d10cf0a0d1a422068d83845df5854bb0e882900e15f6a36912188d90f0b80a41 not found: ID does not exist" Aug 04 11:24:20.055798 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:20.055770 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp"] Aug 04 11:24:20.059638 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:20.059612 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-1cb57d/ipp-route-1cb57d-kserve-7755579ccf-zhjcp"] Aug 04 11:24:21.099464 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:21.099428 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a920567-45c8-43aa-a3a5-9335fcd3b007" path="/var/lib/kubelet/pods/6a920567-45c8-43aa-a3a5-9335fcd3b007/volumes" Aug 04 11:24:48.300147 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:48.300107 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9"] Aug 04 11:24:48.300764 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:48.300393 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" podUID="7d44af1c-5776-4a5e-8f99-c9538db0040b" containerName="istio-proxy" containerID="cri-o://8e4ff3463b17b6ada6721e822bf82defb30bf4750dd3f5e14b731c52a3935315" gracePeriod=30 Aug 04 11:24:53.561331 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:53.561264 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:24:53.678833 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:53.678794 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d44af1c-5776-4a5e-8f99-c9538db0040b-istio-envoy\") pod \"7d44af1c-5776-4a5e-8f99-c9538db0040b\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " Aug 04 11:24:53.679026 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:53.678840 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gr826\" (UniqueName: \"kubernetes.io/projected/7d44af1c-5776-4a5e-8f99-c9538db0040b-kube-api-access-gr826\") pod \"7d44af1c-5776-4a5e-8f99-c9538db0040b\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " Aug 04 11:24:53.679026 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:53.678888 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d44af1c-5776-4a5e-8f99-c9538db0040b-istio-token\") pod \"7d44af1c-5776-4a5e-8f99-c9538db0040b\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " Aug 04 11:24:53.679026 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:53.678927 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d44af1c-5776-4a5e-8f99-c9538db0040b-istio-data\") pod \"7d44af1c-5776-4a5e-8f99-c9538db0040b\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " Aug 04 11:24:53.679026 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:53.678983 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d44af1c-5776-4a5e-8f99-c9538db0040b-workload-socket\") pod \"7d44af1c-5776-4a5e-8f99-c9538db0040b\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " Aug 04 11:24:53.679026 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:53.679024 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d44af1c-5776-4a5e-8f99-c9538db0040b-credential-socket\") pod \"7d44af1c-5776-4a5e-8f99-c9538db0040b\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " Aug 04 11:24:53.679288 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:53.679082 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d44af1c-5776-4a5e-8f99-c9538db0040b-istiod-ca-cert\") pod \"7d44af1c-5776-4a5e-8f99-c9538db0040b\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " Aug 04 11:24:53.679288 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:53.679159 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d44af1c-5776-4a5e-8f99-c9538db0040b-workload-certs\") pod \"7d44af1c-5776-4a5e-8f99-c9538db0040b\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " Aug 04 11:24:53.679288 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:53.679194 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d44af1c-5776-4a5e-8f99-c9538db0040b-istio-podinfo\") pod \"7d44af1c-5776-4a5e-8f99-c9538db0040b\" (UID: \"7d44af1c-5776-4a5e-8f99-c9538db0040b\") " Aug 04 11:24:53.679441 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:53.679278 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d44af1c-5776-4a5e-8f99-c9538db0040b-istio-data" (OuterVolumeSpecName: "istio-data") pod "7d44af1c-5776-4a5e-8f99-c9538db0040b" (UID: "7d44af1c-5776-4a5e-8f99-c9538db0040b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:24:53.679441 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:53.679294 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d44af1c-5776-4a5e-8f99-c9538db0040b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7d44af1c-5776-4a5e-8f99-c9538db0040b" (UID: "7d44af1c-5776-4a5e-8f99-c9538db0040b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:24:53.679683 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:53.679514 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d44af1c-5776-4a5e-8f99-c9538db0040b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7d44af1c-5776-4a5e-8f99-c9538db0040b" (UID: "7d44af1c-5776-4a5e-8f99-c9538db0040b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:24:53.679683 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:53.679595 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d44af1c-5776-4a5e-8f99-c9538db0040b-istio-data\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:24:53.679683 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:53.679618 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d44af1c-5776-4a5e-8f99-c9538db0040b-workload-socket\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:24:53.679864 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:53.679713 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d44af1c-5776-4a5e-8f99-c9538db0040b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7d44af1c-5776-4a5e-8f99-c9538db0040b" (UID: "7d44af1c-5776-4a5e-8f99-c9538db0040b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:24:53.680068 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:53.679997 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d44af1c-5776-4a5e-8f99-c9538db0040b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7d44af1c-5776-4a5e-8f99-c9538db0040b" (UID: "7d44af1c-5776-4a5e-8f99-c9538db0040b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:24:53.681278 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:53.681239 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d44af1c-5776-4a5e-8f99-c9538db0040b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7d44af1c-5776-4a5e-8f99-c9538db0040b" (UID: "7d44af1c-5776-4a5e-8f99-c9538db0040b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:24:53.681408 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:53.681319 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d44af1c-5776-4a5e-8f99-c9538db0040b-kube-api-access-gr826" (OuterVolumeSpecName: "kube-api-access-gr826") pod "7d44af1c-5776-4a5e-8f99-c9538db0040b" (UID: "7d44af1c-5776-4a5e-8f99-c9538db0040b"). InnerVolumeSpecName "kube-api-access-gr826". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:24:53.681408 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:53.681335 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d44af1c-5776-4a5e-8f99-c9538db0040b-istio-token" (OuterVolumeSpecName: "istio-token") pod "7d44af1c-5776-4a5e-8f99-c9538db0040b" (UID: "7d44af1c-5776-4a5e-8f99-c9538db0040b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:24:53.681408 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:53.681383 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d44af1c-5776-4a5e-8f99-c9538db0040b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7d44af1c-5776-4a5e-8f99-c9538db0040b" (UID: "7d44af1c-5776-4a5e-8f99-c9538db0040b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 11:24:53.780848 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:53.780811 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr826\" (UniqueName: \"kubernetes.io/projected/7d44af1c-5776-4a5e-8f99-c9538db0040b-kube-api-access-gr826\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:24:53.780848 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:53.780843 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d44af1c-5776-4a5e-8f99-c9538db0040b-istio-token\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:24:53.780848 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:53.780854 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d44af1c-5776-4a5e-8f99-c9538db0040b-credential-socket\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:24:53.781094 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:53.780862 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d44af1c-5776-4a5e-8f99-c9538db0040b-istiod-ca-cert\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:24:53.781094 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:53.780872 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d44af1c-5776-4a5e-8f99-c9538db0040b-workload-certs\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:24:53.781094 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:53.780880 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d44af1c-5776-4a5e-8f99-c9538db0040b-istio-podinfo\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:24:53.781094 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:53.780888 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d44af1c-5776-4a5e-8f99-c9538db0040b-istio-envoy\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:24:54.205222 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:54.205187 2580 generic.go:356] "Generic (PLEG): container finished" podID="7d44af1c-5776-4a5e-8f99-c9538db0040b" containerID="8e4ff3463b17b6ada6721e822bf82defb30bf4750dd3f5e14b731c52a3935315" exitCode=0 Aug 04 11:24:54.205418 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:54.205249 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" Aug 04 11:24:54.205418 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:54.205285 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" event={"ID":"7d44af1c-5776-4a5e-8f99-c9538db0040b","Type":"ContainerDied","Data":"8e4ff3463b17b6ada6721e822bf82defb30bf4750dd3f5e14b731c52a3935315"} Aug 04 11:24:54.205418 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:54.205343 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9" event={"ID":"7d44af1c-5776-4a5e-8f99-c9538db0040b","Type":"ContainerDied","Data":"ea66e2a14910c6b300de830401e5c5bf9e36c9eb5f0ddbfa2eed2d179ec932f1"} Aug 04 11:24:54.205418 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:54.205366 2580 scope.go:117] "RemoveContainer" containerID="8e4ff3463b17b6ada6721e822bf82defb30bf4750dd3f5e14b731c52a3935315" Aug 04 11:24:54.215201 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:54.215180 2580 scope.go:117] "RemoveContainer" containerID="13b11b983e2c653c028a8d7aba420ab8d936b259656d378ebd4926f4cac5a436" Aug 04 11:24:54.223359 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:54.223338 2580 scope.go:117] "RemoveContainer" containerID="8e4ff3463b17b6ada6721e822bf82defb30bf4750dd3f5e14b731c52a3935315" Aug 04 11:24:54.223618 ip-10-0-142-113 kubenswrapper[2580]: E0804 11:24:54.223598 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e4ff3463b17b6ada6721e822bf82defb30bf4750dd3f5e14b731c52a3935315\": container with ID starting with 8e4ff3463b17b6ada6721e822bf82defb30bf4750dd3f5e14b731c52a3935315 not found: ID does not exist" containerID="8e4ff3463b17b6ada6721e822bf82defb30bf4750dd3f5e14b731c52a3935315" Aug 04 11:24:54.223693 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:54.223625 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e4ff3463b17b6ada6721e822bf82defb30bf4750dd3f5e14b731c52a3935315"} err="failed to get container status \"8e4ff3463b17b6ada6721e822bf82defb30bf4750dd3f5e14b731c52a3935315\": rpc error: code = NotFound desc = could not find container \"8e4ff3463b17b6ada6721e822bf82defb30bf4750dd3f5e14b731c52a3935315\": container with ID starting with 8e4ff3463b17b6ada6721e822bf82defb30bf4750dd3f5e14b731c52a3935315 not found: ID does not exist" Aug 04 11:24:54.223693 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:54.223641 2580 scope.go:117] "RemoveContainer" containerID="13b11b983e2c653c028a8d7aba420ab8d936b259656d378ebd4926f4cac5a436" Aug 04 11:24:54.223861 ip-10-0-142-113 kubenswrapper[2580]: E0804 11:24:54.223846 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13b11b983e2c653c028a8d7aba420ab8d936b259656d378ebd4926f4cac5a436\": container with ID starting with 13b11b983e2c653c028a8d7aba420ab8d936b259656d378ebd4926f4cac5a436 not found: ID does not exist" containerID="13b11b983e2c653c028a8d7aba420ab8d936b259656d378ebd4926f4cac5a436" Aug 04 11:24:54.223917 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:54.223863 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13b11b983e2c653c028a8d7aba420ab8d936b259656d378ebd4926f4cac5a436"} err="failed to get container status \"13b11b983e2c653c028a8d7aba420ab8d936b259656d378ebd4926f4cac5a436\": rpc error: code = NotFound desc = could not find container \"13b11b983e2c653c028a8d7aba420ab8d936b259656d378ebd4926f4cac5a436\": container with ID starting with 13b11b983e2c653c028a8d7aba420ab8d936b259656d378ebd4926f4cac5a436 not found: ID does not exist" Aug 04 11:24:54.228695 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:54.228669 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9"] Aug 04 11:24:54.232608 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:54.232586 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-1cb57d-openshift-default-569df7b7f8-chvl9"] Aug 04 11:24:55.099803 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:24:55.099771 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d44af1c-5776-4a5e-8f99-c9538db0040b" path="/var/lib/kubelet/pods/7d44af1c-5776-4a5e-8f99-c9538db0040b/volumes" Aug 04 11:25:01.439269 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:01.439221 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:25:06.194359 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:06.194314 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj"] Aug 04 11:25:06.194795 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:06.194615 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" podUID="6db0acfe-a199-45d7-8f35-35235cc08cd3" containerName="istio-proxy" containerID="cri-o://4409ad6f3684c3844982eea401ba49638f602cc5948fee172dfff34eb64f320c" gracePeriod=30 Aug 04 11:25:11.452982 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:11.452957 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:25:11.544929 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:11.544835 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db0acfe-a199-45d7-8f35-35235cc08cd3-istiod-ca-cert\") pod \"6db0acfe-a199-45d7-8f35-35235cc08cd3\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " Aug 04 11:25:11.544929 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:11.544902 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db0acfe-a199-45d7-8f35-35235cc08cd3-credential-socket\") pod \"6db0acfe-a199-45d7-8f35-35235cc08cd3\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " Aug 04 11:25:11.545188 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:11.544987 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db0acfe-a199-45d7-8f35-35235cc08cd3-istio-envoy\") pod \"6db0acfe-a199-45d7-8f35-35235cc08cd3\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " Aug 04 11:25:11.545188 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:11.545016 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db0acfe-a199-45d7-8f35-35235cc08cd3-istio-data\") pod \"6db0acfe-a199-45d7-8f35-35235cc08cd3\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " Aug 04 11:25:11.545188 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:11.545044 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db0acfe-a199-45d7-8f35-35235cc08cd3-istio-podinfo\") pod \"6db0acfe-a199-45d7-8f35-35235cc08cd3\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " Aug 04 11:25:11.545188 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:11.545079 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db0acfe-a199-45d7-8f35-35235cc08cd3-workload-certs\") pod \"6db0acfe-a199-45d7-8f35-35235cc08cd3\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " Aug 04 11:25:11.545188 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:11.545109 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db0acfe-a199-45d7-8f35-35235cc08cd3-istio-token\") pod \"6db0acfe-a199-45d7-8f35-35235cc08cd3\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " Aug 04 11:25:11.545458 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:11.545236 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db0acfe-a199-45d7-8f35-35235cc08cd3-istio-data" (OuterVolumeSpecName: "istio-data") pod "6db0acfe-a199-45d7-8f35-35235cc08cd3" (UID: "6db0acfe-a199-45d7-8f35-35235cc08cd3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:25:11.545458 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:11.545273 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db0acfe-a199-45d7-8f35-35235cc08cd3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6db0acfe-a199-45d7-8f35-35235cc08cd3" (UID: "6db0acfe-a199-45d7-8f35-35235cc08cd3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:25:11.545458 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:11.545287 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db0acfe-a199-45d7-8f35-35235cc08cd3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6db0acfe-a199-45d7-8f35-35235cc08cd3" (UID: "6db0acfe-a199-45d7-8f35-35235cc08cd3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:25:11.545458 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:11.545296 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gm6x7\" (UniqueName: \"kubernetes.io/projected/6db0acfe-a199-45d7-8f35-35235cc08cd3-kube-api-access-gm6x7\") pod \"6db0acfe-a199-45d7-8f35-35235cc08cd3\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " Aug 04 11:25:11.545458 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:11.545352 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db0acfe-a199-45d7-8f35-35235cc08cd3-workload-socket\") pod \"6db0acfe-a199-45d7-8f35-35235cc08cd3\" (UID: \"6db0acfe-a199-45d7-8f35-35235cc08cd3\") " Aug 04 11:25:11.545875 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:11.545478 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db0acfe-a199-45d7-8f35-35235cc08cd3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6db0acfe-a199-45d7-8f35-35235cc08cd3" (UID: "6db0acfe-a199-45d7-8f35-35235cc08cd3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:25:11.545875 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:11.545583 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db0acfe-a199-45d7-8f35-35235cc08cd3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6db0acfe-a199-45d7-8f35-35235cc08cd3" (UID: "6db0acfe-a199-45d7-8f35-35235cc08cd3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:25:11.545875 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:11.545774 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db0acfe-a199-45d7-8f35-35235cc08cd3-istio-data\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:25:11.545875 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:11.545794 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db0acfe-a199-45d7-8f35-35235cc08cd3-workload-certs\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:25:11.545875 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:11.545808 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db0acfe-a199-45d7-8f35-35235cc08cd3-workload-socket\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:25:11.545875 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:11.545822 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db0acfe-a199-45d7-8f35-35235cc08cd3-istiod-ca-cert\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:25:11.546217 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:11.545901 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db0acfe-a199-45d7-8f35-35235cc08cd3-credential-socket\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:25:11.547474 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:11.547446 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db0acfe-a199-45d7-8f35-35235cc08cd3-istio-token" (OuterVolumeSpecName: "istio-token") pod "6db0acfe-a199-45d7-8f35-35235cc08cd3" (UID: "6db0acfe-a199-45d7-8f35-35235cc08cd3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:25:11.547474 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:11.547454 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db0acfe-a199-45d7-8f35-35235cc08cd3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6db0acfe-a199-45d7-8f35-35235cc08cd3" (UID: "6db0acfe-a199-45d7-8f35-35235cc08cd3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:25:11.547651 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:11.547470 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6db0acfe-a199-45d7-8f35-35235cc08cd3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6db0acfe-a199-45d7-8f35-35235cc08cd3" (UID: "6db0acfe-a199-45d7-8f35-35235cc08cd3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 11:25:11.547651 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:11.547577 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db0acfe-a199-45d7-8f35-35235cc08cd3-kube-api-access-gm6x7" (OuterVolumeSpecName: "kube-api-access-gm6x7") pod "6db0acfe-a199-45d7-8f35-35235cc08cd3" (UID: "6db0acfe-a199-45d7-8f35-35235cc08cd3"). InnerVolumeSpecName "kube-api-access-gm6x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:25:11.647385 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:11.647345 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db0acfe-a199-45d7-8f35-35235cc08cd3-istio-envoy\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:25:11.647385 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:11.647377 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db0acfe-a199-45d7-8f35-35235cc08cd3-istio-podinfo\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:25:11.647385 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:11.647387 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db0acfe-a199-45d7-8f35-35235cc08cd3-istio-token\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:25:11.647385 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:11.647396 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm6x7\" (UniqueName: \"kubernetes.io/projected/6db0acfe-a199-45d7-8f35-35235cc08cd3-kube-api-access-gm6x7\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:25:12.301814 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:12.301775 2580 generic.go:356] "Generic (PLEG): container finished" podID="6db0acfe-a199-45d7-8f35-35235cc08cd3" containerID="4409ad6f3684c3844982eea401ba49638f602cc5948fee172dfff34eb64f320c" exitCode=0 Aug 04 11:25:12.302020 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:12.301828 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" event={"ID":"6db0acfe-a199-45d7-8f35-35235cc08cd3","Type":"ContainerDied","Data":"4409ad6f3684c3844982eea401ba49638f602cc5948fee172dfff34eb64f320c"} Aug 04 11:25:12.302020 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:12.301840 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" Aug 04 11:25:12.302020 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:12.301925 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj" event={"ID":"6db0acfe-a199-45d7-8f35-35235cc08cd3","Type":"ContainerDied","Data":"b201051a4b1fd846c291386fe94386d870789f7aac36b18e7d381c6211affdd8"} Aug 04 11:25:12.302020 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:12.301959 2580 scope.go:117] "RemoveContainer" containerID="4409ad6f3684c3844982eea401ba49638f602cc5948fee172dfff34eb64f320c" Aug 04 11:25:12.311051 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:12.311030 2580 scope.go:117] "RemoveContainer" containerID="bf20c6b4fdfba70ed5636f158a3bbadd92d876badfe1375a7fb94eea2a27bb75" Aug 04 11:25:12.319164 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:12.319014 2580 scope.go:117] "RemoveContainer" containerID="4409ad6f3684c3844982eea401ba49638f602cc5948fee172dfff34eb64f320c" Aug 04 11:25:12.319297 ip-10-0-142-113 kubenswrapper[2580]: E0804 11:25:12.319277 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4409ad6f3684c3844982eea401ba49638f602cc5948fee172dfff34eb64f320c\": container with ID starting with 4409ad6f3684c3844982eea401ba49638f602cc5948fee172dfff34eb64f320c not found: ID does not exist" containerID="4409ad6f3684c3844982eea401ba49638f602cc5948fee172dfff34eb64f320c" Aug 04 11:25:12.319345 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:12.319305 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4409ad6f3684c3844982eea401ba49638f602cc5948fee172dfff34eb64f320c"} err="failed to get container status \"4409ad6f3684c3844982eea401ba49638f602cc5948fee172dfff34eb64f320c\": rpc error: code = NotFound desc = could not find container \"4409ad6f3684c3844982eea401ba49638f602cc5948fee172dfff34eb64f320c\": container with ID starting with 4409ad6f3684c3844982eea401ba49638f602cc5948fee172dfff34eb64f320c not found: ID does not exist" Aug 04 11:25:12.319345 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:12.319322 2580 scope.go:117] "RemoveContainer" containerID="bf20c6b4fdfba70ed5636f158a3bbadd92d876badfe1375a7fb94eea2a27bb75" Aug 04 11:25:12.319530 ip-10-0-142-113 kubenswrapper[2580]: E0804 11:25:12.319514 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf20c6b4fdfba70ed5636f158a3bbadd92d876badfe1375a7fb94eea2a27bb75\": container with ID starting with bf20c6b4fdfba70ed5636f158a3bbadd92d876badfe1375a7fb94eea2a27bb75 not found: ID does not exist" containerID="bf20c6b4fdfba70ed5636f158a3bbadd92d876badfe1375a7fb94eea2a27bb75" Aug 04 11:25:12.319585 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:12.319535 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf20c6b4fdfba70ed5636f158a3bbadd92d876badfe1375a7fb94eea2a27bb75"} err="failed to get container status \"bf20c6b4fdfba70ed5636f158a3bbadd92d876badfe1375a7fb94eea2a27bb75\": rpc error: code = NotFound desc = could not find container \"bf20c6b4fdfba70ed5636f158a3bbadd92d876badfe1375a7fb94eea2a27bb75\": container with ID starting with bf20c6b4fdfba70ed5636f158a3bbadd92d876badfe1375a7fb94eea2a27bb75 not found: ID does not exist" Aug 04 11:25:12.324785 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:12.324763 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj"] Aug 04 11:25:12.328516 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:12.328495 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-651a7a-openshift-default-74d6994b9d-7qqkj"] Aug 04 11:25:13.099875 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:13.099839 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6db0acfe-a199-45d7-8f35-35235cc08cd3" path="/var/lib/kubelet/pods/6db0acfe-a199-45d7-8f35-35235cc08cd3/volumes" Aug 04 11:25:19.436586 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:19.436534 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:25:24.179692 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:24.179659 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg"] Aug 04 11:25:24.180085 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:24.179875 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" podUID="d81863b6-bc8d-403a-9399-dff1f20d4c59" containerName="istio-proxy" containerID="cri-o://f33484a83fccb724ae79de0ab2742e4e940b2c8ed22952c6ff3e0db5e2150b24" gracePeriod=30 Aug 04 11:25:29.385923 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:29.385856 2580 generic.go:356] "Generic (PLEG): container finished" podID="d81863b6-bc8d-403a-9399-dff1f20d4c59" containerID="f33484a83fccb724ae79de0ab2742e4e940b2c8ed22952c6ff3e0db5e2150b24" exitCode=0 Aug 04 11:25:29.386313 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:29.385924 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" event={"ID":"d81863b6-bc8d-403a-9399-dff1f20d4c59","Type":"ContainerDied","Data":"f33484a83fccb724ae79de0ab2742e4e940b2c8ed22952c6ff3e0db5e2150b24"} Aug 04 11:25:29.386313 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:29.386013 2580 scope.go:117] "RemoveContainer" containerID="3ebae1001e6057b74a5a1ce893bd2bb50f9503954b3097e2bb0e899a7e24609d" Aug 04 11:25:29.442870 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:29.442845 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:25:29.486153 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:29.486071 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81863b6-bc8d-403a-9399-dff1f20d4c59-istiod-ca-cert\") pod \"d81863b6-bc8d-403a-9399-dff1f20d4c59\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " Aug 04 11:25:29.486153 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:29.486127 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81863b6-bc8d-403a-9399-dff1f20d4c59-credential-socket\") pod \"d81863b6-bc8d-403a-9399-dff1f20d4c59\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " Aug 04 11:25:29.486386 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:29.486162 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24sdb\" (UniqueName: \"kubernetes.io/projected/d81863b6-bc8d-403a-9399-dff1f20d4c59-kube-api-access-24sdb\") pod \"d81863b6-bc8d-403a-9399-dff1f20d4c59\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " Aug 04 11:25:29.486386 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:29.486309 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81863b6-bc8d-403a-9399-dff1f20d4c59-workload-certs\") pod \"d81863b6-bc8d-403a-9399-dff1f20d4c59\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " Aug 04 11:25:29.486386 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:29.486353 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81863b6-bc8d-403a-9399-dff1f20d4c59-workload-socket\") pod \"d81863b6-bc8d-403a-9399-dff1f20d4c59\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " Aug 04 11:25:29.486386 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:29.486384 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81863b6-bc8d-403a-9399-dff1f20d4c59-istio-token\") pod \"d81863b6-bc8d-403a-9399-dff1f20d4c59\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " Aug 04 11:25:29.486617 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:29.486409 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81863b6-bc8d-403a-9399-dff1f20d4c59-istio-podinfo\") pod \"d81863b6-bc8d-403a-9399-dff1f20d4c59\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " Aug 04 11:25:29.486617 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:29.486455 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81863b6-bc8d-403a-9399-dff1f20d4c59-istio-data\") pod \"d81863b6-bc8d-403a-9399-dff1f20d4c59\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " Aug 04 11:25:29.486617 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:29.486448 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81863b6-bc8d-403a-9399-dff1f20d4c59-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d81863b6-bc8d-403a-9399-dff1f20d4c59" (UID: "d81863b6-bc8d-403a-9399-dff1f20d4c59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:25:29.486617 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:29.486480 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81863b6-bc8d-403a-9399-dff1f20d4c59-istio-envoy\") pod \"d81863b6-bc8d-403a-9399-dff1f20d4c59\" (UID: \"d81863b6-bc8d-403a-9399-dff1f20d4c59\") " Aug 04 11:25:29.486617 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:29.486575 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81863b6-bc8d-403a-9399-dff1f20d4c59-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d81863b6-bc8d-403a-9399-dff1f20d4c59" (UID: "d81863b6-bc8d-403a-9399-dff1f20d4c59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:25:29.486871 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:29.486620 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d81863b6-bc8d-403a-9399-dff1f20d4c59-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d81863b6-bc8d-403a-9399-dff1f20d4c59" (UID: "d81863b6-bc8d-403a-9399-dff1f20d4c59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:25:29.486871 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:29.486780 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81863b6-bc8d-403a-9399-dff1f20d4c59-istio-data" (OuterVolumeSpecName: "istio-data") pod "d81863b6-bc8d-403a-9399-dff1f20d4c59" (UID: "d81863b6-bc8d-403a-9399-dff1f20d4c59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:25:29.486871 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:29.486839 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81863b6-bc8d-403a-9399-dff1f20d4c59-istiod-ca-cert\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:25:29.486871 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:29.486858 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81863b6-bc8d-403a-9399-dff1f20d4c59-credential-socket\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:25:29.487021 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:29.486874 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81863b6-bc8d-403a-9399-dff1f20d4c59-workload-certs\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:25:29.487021 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:29.486933 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81863b6-bc8d-403a-9399-dff1f20d4c59-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d81863b6-bc8d-403a-9399-dff1f20d4c59" (UID: "d81863b6-bc8d-403a-9399-dff1f20d4c59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:25:29.488651 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:29.488625 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d81863b6-bc8d-403a-9399-dff1f20d4c59-kube-api-access-24sdb" (OuterVolumeSpecName: "kube-api-access-24sdb") pod "d81863b6-bc8d-403a-9399-dff1f20d4c59" (UID: "d81863b6-bc8d-403a-9399-dff1f20d4c59"). InnerVolumeSpecName "kube-api-access-24sdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:25:29.488912 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:29.488890 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81863b6-bc8d-403a-9399-dff1f20d4c59-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d81863b6-bc8d-403a-9399-dff1f20d4c59" (UID: "d81863b6-bc8d-403a-9399-dff1f20d4c59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:25:29.489165 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:29.489145 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d81863b6-bc8d-403a-9399-dff1f20d4c59-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d81863b6-bc8d-403a-9399-dff1f20d4c59" (UID: "d81863b6-bc8d-403a-9399-dff1f20d4c59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 11:25:29.489223 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:29.489145 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d81863b6-bc8d-403a-9399-dff1f20d4c59-istio-token" (OuterVolumeSpecName: "istio-token") pod "d81863b6-bc8d-403a-9399-dff1f20d4c59" (UID: "d81863b6-bc8d-403a-9399-dff1f20d4c59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:25:29.587947 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:29.587903 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24sdb\" (UniqueName: \"kubernetes.io/projected/d81863b6-bc8d-403a-9399-dff1f20d4c59-kube-api-access-24sdb\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:25:29.587947 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:29.587939 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81863b6-bc8d-403a-9399-dff1f20d4c59-workload-socket\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:25:29.587947 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:29.587948 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81863b6-bc8d-403a-9399-dff1f20d4c59-istio-token\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:25:29.587947 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:29.587958 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81863b6-bc8d-403a-9399-dff1f20d4c59-istio-podinfo\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:25:29.588206 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:29.587966 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81863b6-bc8d-403a-9399-dff1f20d4c59-istio-data\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:25:29.588206 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:29.587975 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81863b6-bc8d-403a-9399-dff1f20d4c59-istio-envoy\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:25:30.391507 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:30.391475 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" Aug 04 11:25:30.391507 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:30.391480 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg" event={"ID":"d81863b6-bc8d-403a-9399-dff1f20d4c59","Type":"ContainerDied","Data":"daa4255ab5f413955ca9245989264c35884c86ec4cc0c7a7621f487cdc8007d7"} Aug 04 11:25:30.392017 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:30.391535 2580 scope.go:117] "RemoveContainer" containerID="f33484a83fccb724ae79de0ab2742e4e940b2c8ed22952c6ff3e0db5e2150b24" Aug 04 11:25:30.414583 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:30.414537 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg"] Aug 04 11:25:30.419006 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:30.418985 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-bc4a78-openshift-default-558d94c688-f7qxg"] Aug 04 11:25:31.100176 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:31.100138 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d81863b6-bc8d-403a-9399-dff1f20d4c59" path="/var/lib/kubelet/pods/d81863b6-bc8d-403a-9399-dff1f20d4c59/volumes" Aug 04 11:25:37.130746 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:37.130714 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jmqqj"] Aug 04 11:25:42.025211 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:42.025158 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w"] Aug 04 11:25:42.025771 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:42.025434 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" podUID="a4414c58-5988-4ed1-a563-403975e8e0d9" containerName="istio-proxy" containerID="cri-o://2282034771a0f99f4e1b88a32b2ef1a548bddf2f8af710d5efb224c0cdf6b600" gracePeriod=30 Aug 04 11:25:47.289000 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.288975 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:25:47.346653 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.346547 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4414c58-5988-4ed1-a563-403975e8e0d9-credential-socket\") pod \"a4414c58-5988-4ed1-a563-403975e8e0d9\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " Aug 04 11:25:47.346653 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.346627 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nq9z5\" (UniqueName: \"kubernetes.io/projected/a4414c58-5988-4ed1-a563-403975e8e0d9-kube-api-access-nq9z5\") pod \"a4414c58-5988-4ed1-a563-403975e8e0d9\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " Aug 04 11:25:47.346847 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.346664 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4414c58-5988-4ed1-a563-403975e8e0d9-istio-data\") pod \"a4414c58-5988-4ed1-a563-403975e8e0d9\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " Aug 04 11:25:47.346847 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.346688 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4414c58-5988-4ed1-a563-403975e8e0d9-workload-socket\") pod \"a4414c58-5988-4ed1-a563-403975e8e0d9\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " Aug 04 11:25:47.346847 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.346728 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4414c58-5988-4ed1-a563-403975e8e0d9-istiod-ca-cert\") pod \"a4414c58-5988-4ed1-a563-403975e8e0d9\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " Aug 04 11:25:47.346847 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.346789 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4414c58-5988-4ed1-a563-403975e8e0d9-istio-envoy\") pod \"a4414c58-5988-4ed1-a563-403975e8e0d9\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " Aug 04 11:25:47.346847 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.346825 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4414c58-5988-4ed1-a563-403975e8e0d9-workload-certs\") pod \"a4414c58-5988-4ed1-a563-403975e8e0d9\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " Aug 04 11:25:47.347097 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.346859 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4414c58-5988-4ed1-a563-403975e8e0d9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a4414c58-5988-4ed1-a563-403975e8e0d9" (UID: "a4414c58-5988-4ed1-a563-403975e8e0d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:25:47.347097 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.346882 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4414c58-5988-4ed1-a563-403975e8e0d9-istio-token\") pod \"a4414c58-5988-4ed1-a563-403975e8e0d9\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " Aug 04 11:25:47.347097 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.346944 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4414c58-5988-4ed1-a563-403975e8e0d9-istio-data" (OuterVolumeSpecName: "istio-data") pod "a4414c58-5988-4ed1-a563-403975e8e0d9" (UID: "a4414c58-5988-4ed1-a563-403975e8e0d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:25:47.347097 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.346976 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4414c58-5988-4ed1-a563-403975e8e0d9-istio-podinfo\") pod \"a4414c58-5988-4ed1-a563-403975e8e0d9\" (UID: \"a4414c58-5988-4ed1-a563-403975e8e0d9\") " Aug 04 11:25:47.347274 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.347124 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4414c58-5988-4ed1-a563-403975e8e0d9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a4414c58-5988-4ed1-a563-403975e8e0d9" (UID: "a4414c58-5988-4ed1-a563-403975e8e0d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:25:47.347274 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.347243 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4414c58-5988-4ed1-a563-403975e8e0d9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a4414c58-5988-4ed1-a563-403975e8e0d9" (UID: "a4414c58-5988-4ed1-a563-403975e8e0d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:25:47.347373 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.347299 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4414c58-5988-4ed1-a563-403975e8e0d9-credential-socket\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:25:47.347373 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.347314 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4414c58-5988-4ed1-a563-403975e8e0d9-istio-data\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:25:47.347373 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.347326 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4414c58-5988-4ed1-a563-403975e8e0d9-workload-certs\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:25:47.347660 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.347578 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4414c58-5988-4ed1-a563-403975e8e0d9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a4414c58-5988-4ed1-a563-403975e8e0d9" (UID: "a4414c58-5988-4ed1-a563-403975e8e0d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:25:47.349229 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.349200 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4414c58-5988-4ed1-a563-403975e8e0d9-kube-api-access-nq9z5" (OuterVolumeSpecName: "kube-api-access-nq9z5") pod "a4414c58-5988-4ed1-a563-403975e8e0d9" (UID: "a4414c58-5988-4ed1-a563-403975e8e0d9"). InnerVolumeSpecName "kube-api-access-nq9z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:25:47.349506 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.349481 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4414c58-5988-4ed1-a563-403975e8e0d9-istio-token" (OuterVolumeSpecName: "istio-token") pod "a4414c58-5988-4ed1-a563-403975e8e0d9" (UID: "a4414c58-5988-4ed1-a563-403975e8e0d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:25:47.349506 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.349479 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4414c58-5988-4ed1-a563-403975e8e0d9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a4414c58-5988-4ed1-a563-403975e8e0d9" (UID: "a4414c58-5988-4ed1-a563-403975e8e0d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:25:47.349665 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.349545 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4414c58-5988-4ed1-a563-403975e8e0d9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a4414c58-5988-4ed1-a563-403975e8e0d9" (UID: "a4414c58-5988-4ed1-a563-403975e8e0d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 11:25:47.448732 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.448696 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4414c58-5988-4ed1-a563-403975e8e0d9-istio-token\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:25:47.448732 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.448723 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4414c58-5988-4ed1-a563-403975e8e0d9-istio-podinfo\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:25:47.448732 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.448732 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq9z5\" (UniqueName: \"kubernetes.io/projected/a4414c58-5988-4ed1-a563-403975e8e0d9-kube-api-access-nq9z5\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:25:47.448732 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.448742 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4414c58-5988-4ed1-a563-403975e8e0d9-workload-socket\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:25:47.448992 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.448752 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4414c58-5988-4ed1-a563-403975e8e0d9-istiod-ca-cert\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:25:47.448992 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.448760 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4414c58-5988-4ed1-a563-403975e8e0d9-istio-envoy\") on node \"ip-10-0-142-113.ec2.internal\" DevicePath \"\"" Aug 04 11:25:47.471882 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.471849 2580 generic.go:356] "Generic (PLEG): container finished" podID="a4414c58-5988-4ed1-a563-403975e8e0d9" containerID="2282034771a0f99f4e1b88a32b2ef1a548bddf2f8af710d5efb224c0cdf6b600" exitCode=0 Aug 04 11:25:47.472047 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.471899 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" Aug 04 11:25:47.472047 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.471930 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" event={"ID":"a4414c58-5988-4ed1-a563-403975e8e0d9","Type":"ContainerDied","Data":"2282034771a0f99f4e1b88a32b2ef1a548bddf2f8af710d5efb224c0cdf6b600"} Aug 04 11:25:47.472047 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.471990 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w" event={"ID":"a4414c58-5988-4ed1-a563-403975e8e0d9","Type":"ContainerDied","Data":"91687d50a92fe41b5138a51de11b341ebf51c345d3356bc0deb0736f3e4c2193"} Aug 04 11:25:47.472047 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.472020 2580 scope.go:117] "RemoveContainer" containerID="2282034771a0f99f4e1b88a32b2ef1a548bddf2f8af710d5efb224c0cdf6b600" Aug 04 11:25:47.481504 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.481476 2580 scope.go:117] "RemoveContainer" containerID="6799fd0b9e8dac093b6940309059a886ec0cf241e83de178b86fd1ab18ca19e8" Aug 04 11:25:47.489532 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.489513 2580 scope.go:117] "RemoveContainer" containerID="2282034771a0f99f4e1b88a32b2ef1a548bddf2f8af710d5efb224c0cdf6b600" Aug 04 11:25:47.489816 ip-10-0-142-113 kubenswrapper[2580]: E0804 11:25:47.489798 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2282034771a0f99f4e1b88a32b2ef1a548bddf2f8af710d5efb224c0cdf6b600\": container with ID starting with 2282034771a0f99f4e1b88a32b2ef1a548bddf2f8af710d5efb224c0cdf6b600 not found: ID does not exist" containerID="2282034771a0f99f4e1b88a32b2ef1a548bddf2f8af710d5efb224c0cdf6b600" Aug 04 11:25:47.489875 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.489826 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2282034771a0f99f4e1b88a32b2ef1a548bddf2f8af710d5efb224c0cdf6b600"} err="failed to get container status \"2282034771a0f99f4e1b88a32b2ef1a548bddf2f8af710d5efb224c0cdf6b600\": rpc error: code = NotFound desc = could not find container \"2282034771a0f99f4e1b88a32b2ef1a548bddf2f8af710d5efb224c0cdf6b600\": container with ID starting with 2282034771a0f99f4e1b88a32b2ef1a548bddf2f8af710d5efb224c0cdf6b600 not found: ID does not exist" Aug 04 11:25:47.489875 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.489843 2580 scope.go:117] "RemoveContainer" containerID="6799fd0b9e8dac093b6940309059a886ec0cf241e83de178b86fd1ab18ca19e8" Aug 04 11:25:47.490074 ip-10-0-142-113 kubenswrapper[2580]: E0804 11:25:47.490060 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6799fd0b9e8dac093b6940309059a886ec0cf241e83de178b86fd1ab18ca19e8\": container with ID starting with 6799fd0b9e8dac093b6940309059a886ec0cf241e83de178b86fd1ab18ca19e8 not found: ID does not exist" containerID="6799fd0b9e8dac093b6940309059a886ec0cf241e83de178b86fd1ab18ca19e8" Aug 04 11:25:47.490111 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.490078 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6799fd0b9e8dac093b6940309059a886ec0cf241e83de178b86fd1ab18ca19e8"} err="failed to get container status \"6799fd0b9e8dac093b6940309059a886ec0cf241e83de178b86fd1ab18ca19e8\": rpc error: code = NotFound desc = could not find container \"6799fd0b9e8dac093b6940309059a886ec0cf241e83de178b86fd1ab18ca19e8\": container with ID starting with 6799fd0b9e8dac093b6940309059a886ec0cf241e83de178b86fd1ab18ca19e8 not found: ID does not exist" Aug 04 11:25:47.494075 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.494053 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w"] Aug 04 11:25:47.498019 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:47.497995 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-460cac-openshift-default-556754cbcb-r2b6w"] Aug 04 11:25:48.259436 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:48.259396 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-wnv4x_a4a17547-6158-4123-bc5f-39d76f32974a/manager/0.log" Aug 04 11:25:48.382302 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:48.382272 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-kl2rb_7b6a041d-9d50-4a1e-bbfa-527e5859d8cd/manager/0.log" Aug 04 11:25:48.628165 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:48.628140 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vq67x_f44b71f1-14e3-420e-b826-0705280ac08f/server/0.log" Aug 04 11:25:48.753782 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:48.753751 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-hjxw2_b36de69a-8276-4697-a3b3-194f302385d7/manager/0.log" Aug 04 11:25:48.867112 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:48.867085 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-598d67d848-6jswn_6765da31-9d18-4e54-bdc9-886576264de3/copy-manifests/0.log" Aug 04 11:25:48.874517 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:48.874488 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-598d67d848-6jswn_6765da31-9d18-4e54-bdc9-886576264de3/manager/0.log" Aug 04 11:25:49.099867 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:49.099788 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4414c58-5988-4ed1-a563-403975e8e0d9" path="/var/lib/kubelet/pods/a4414c58-5988-4ed1-a563-403975e8e0d9/volumes" Aug 04 11:25:50.860419 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:50.860389 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-gwn2m_b3f3b35a-3c47-4410-afd5-671953692c16/manager/0.log" Aug 04 11:25:50.967031 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:50.966998 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-4npks_2893cf39-ba4a-49bc-9660-c3123ae361ba/manager/0.log" Aug 04 11:25:51.216113 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:51.216035 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5b65f94d4c-h56g4_96ffd714-48bf-432c-8ce1-112fb49eea4a/manager/0.log" Aug 04 11:25:51.332771 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:51.332741 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-jmqqj_b3b6de8b-8543-42eb-9292-5a925d23fe0f/limitador/0.log" Aug 04 11:25:51.470268 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:51.470181 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-qkfqh_1e55781b-f662-4806-8b91-fe7ab66de387/manager/0.log" Aug 04 11:25:51.817535 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:51.817454 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv_7ee19c9c-fbf9-4f49-8520-0aedeee66aa6/istio-proxy/0.log" Aug 04 11:25:52.180833 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:52.180796 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bd95bfc5-zdsld_b112fb0c-d3f1-43c6-bef8-607a04229e08/kube-auth-proxy/0.log" Aug 04 11:25:52.300782 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:52.300745 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tqq7s_f93fb736-f6f2-4b03-9821-4036c36e2f72/istio-proxy/1.log" Aug 04 11:25:53.451813 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:53.451777 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-tnqkq_5b308251-1c5d-4a9a-adc3-a741a85c55f9/main/0.log" Aug 04 11:25:59.382209 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.382173 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qfz6h/must-gather-dlqzg"] Aug 04 11:25:59.382604 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.382485 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d44af1c-5776-4a5e-8f99-c9538db0040b" containerName="istio-proxy" Aug 04 11:25:59.382604 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.382495 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d44af1c-5776-4a5e-8f99-c9538db0040b" containerName="istio-proxy" Aug 04 11:25:59.382604 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.382513 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4414c58-5988-4ed1-a563-403975e8e0d9" containerName="istio-proxy" Aug 04 11:25:59.382604 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.382518 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4414c58-5988-4ed1-a563-403975e8e0d9" containerName="istio-proxy" Aug 04 11:25:59.382604 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.382539 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6db0acfe-a199-45d7-8f35-35235cc08cd3" containerName="istio-proxy" Aug 04 11:25:59.382604 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.382545 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6db0acfe-a199-45d7-8f35-35235cc08cd3" containerName="istio-proxy" Aug 04 11:25:59.382604 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.382551 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a920567-45c8-43aa-a3a5-9335fcd3b007" containerName="main" Aug 04 11:25:59.382604 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.382556 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a920567-45c8-43aa-a3a5-9335fcd3b007" containerName="main" Aug 04 11:25:59.382604 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.382579 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d81863b6-bc8d-403a-9399-dff1f20d4c59" containerName="istio-proxy" Aug 04 11:25:59.382604 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.382584 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d81863b6-bc8d-403a-9399-dff1f20d4c59" containerName="istio-proxy" Aug 04 11:25:59.382604 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.382590 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d81863b6-bc8d-403a-9399-dff1f20d4c59" containerName="istio-proxy" Aug 04 11:25:59.382604 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.382595 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d81863b6-bc8d-403a-9399-dff1f20d4c59" containerName="istio-proxy" Aug 04 11:25:59.382604 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.382601 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6db0acfe-a199-45d7-8f35-35235cc08cd3" containerName="istio-proxy" Aug 04 11:25:59.382604 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.382607 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6db0acfe-a199-45d7-8f35-35235cc08cd3" containerName="istio-proxy" Aug 04 11:25:59.383049 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.382652 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d44af1c-5776-4a5e-8f99-c9538db0040b" containerName="istio-proxy" Aug 04 11:25:59.383049 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.382658 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d81863b6-bc8d-403a-9399-dff1f20d4c59" containerName="istio-proxy" Aug 04 11:25:59.383049 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.382664 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6db0acfe-a199-45d7-8f35-35235cc08cd3" containerName="istio-proxy" Aug 04 11:25:59.383049 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.382670 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a920567-45c8-43aa-a3a5-9335fcd3b007" containerName="main" Aug 04 11:25:59.383049 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.382676 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4414c58-5988-4ed1-a563-403975e8e0d9" containerName="istio-proxy" Aug 04 11:25:59.383049 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.382681 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4414c58-5988-4ed1-a563-403975e8e0d9" containerName="istio-proxy" Aug 04 11:25:59.383049 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.382726 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4414c58-5988-4ed1-a563-403975e8e0d9" containerName="istio-proxy" Aug 04 11:25:59.383049 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.382731 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4414c58-5988-4ed1-a563-403975e8e0d9" containerName="istio-proxy" Aug 04 11:25:59.383049 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.382742 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d44af1c-5776-4a5e-8f99-c9538db0040b" containerName="istio-proxy" Aug 04 11:25:59.383049 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.382747 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d44af1c-5776-4a5e-8f99-c9538db0040b" containerName="istio-proxy" Aug 04 11:25:59.383049 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.382786 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d81863b6-bc8d-403a-9399-dff1f20d4c59" containerName="istio-proxy" Aug 04 11:25:59.383049 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.382791 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6db0acfe-a199-45d7-8f35-35235cc08cd3" containerName="istio-proxy" Aug 04 11:25:59.383049 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.382800 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d44af1c-5776-4a5e-8f99-c9538db0040b" containerName="istio-proxy" Aug 04 11:25:59.385675 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.385658 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qfz6h/must-gather-dlqzg" Aug 04 11:25:59.388546 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.388523 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qfz6h\"/\"default-dockercfg-67krk\"" Aug 04 11:25:59.389878 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.389863 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qfz6h\"/\"kube-root-ca.crt\"" Aug 04 11:25:59.389957 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.389874 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qfz6h\"/\"openshift-service-ca.crt\"" Aug 04 11:25:59.393471 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.393444 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qfz6h/must-gather-dlqzg"] Aug 04 11:25:59.456991 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.456949 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phvx8\" (UniqueName: \"kubernetes.io/projected/26649b35-b51b-4de0-b60d-95cd8ce67d9a-kube-api-access-phvx8\") pod \"must-gather-dlqzg\" (UID: \"26649b35-b51b-4de0-b60d-95cd8ce67d9a\") " pod="openshift-must-gather-qfz6h/must-gather-dlqzg" Aug 04 11:25:59.456991 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.456997 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/26649b35-b51b-4de0-b60d-95cd8ce67d9a-must-gather-output\") pod \"must-gather-dlqzg\" (UID: \"26649b35-b51b-4de0-b60d-95cd8ce67d9a\") " pod="openshift-must-gather-qfz6h/must-gather-dlqzg" Aug 04 11:25:59.558277 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.558234 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phvx8\" (UniqueName: \"kubernetes.io/projected/26649b35-b51b-4de0-b60d-95cd8ce67d9a-kube-api-access-phvx8\") pod \"must-gather-dlqzg\" (UID: \"26649b35-b51b-4de0-b60d-95cd8ce67d9a\") " pod="openshift-must-gather-qfz6h/must-gather-dlqzg" Aug 04 11:25:59.558464 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.558288 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/26649b35-b51b-4de0-b60d-95cd8ce67d9a-must-gather-output\") pod \"must-gather-dlqzg\" (UID: \"26649b35-b51b-4de0-b60d-95cd8ce67d9a\") " pod="openshift-must-gather-qfz6h/must-gather-dlqzg" Aug 04 11:25:59.558654 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.558634 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/26649b35-b51b-4de0-b60d-95cd8ce67d9a-must-gather-output\") pod \"must-gather-dlqzg\" (UID: \"26649b35-b51b-4de0-b60d-95cd8ce67d9a\") " pod="openshift-must-gather-qfz6h/must-gather-dlqzg" Aug 04 11:25:59.567774 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.567747 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phvx8\" (UniqueName: \"kubernetes.io/projected/26649b35-b51b-4de0-b60d-95cd8ce67d9a-kube-api-access-phvx8\") pod \"must-gather-dlqzg\" (UID: \"26649b35-b51b-4de0-b60d-95cd8ce67d9a\") " pod="openshift-must-gather-qfz6h/must-gather-dlqzg" Aug 04 11:25:59.695237 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.695162 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qfz6h/must-gather-dlqzg" Aug 04 11:25:59.825678 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.825620 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qfz6h/must-gather-dlqzg"] Aug 04 11:25:59.827743 ip-10-0-142-113 kubenswrapper[2580]: W0804 11:25:59.827716 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26649b35_b51b_4de0_b60d_95cd8ce67d9a.slice/crio-232e758b6ba91ae672ef3ad42fd70a5aa6396091cd4068a450909006d4c86d78 WatchSource:0}: Error finding container 232e758b6ba91ae672ef3ad42fd70a5aa6396091cd4068a450909006d4c86d78: Status 404 returned error can't find the container with id 232e758b6ba91ae672ef3ad42fd70a5aa6396091cd4068a450909006d4c86d78 Aug 04 11:25:59.829681 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:25:59.829661 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 11:26:00.536378 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:00.536336 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qfz6h/must-gather-dlqzg" event={"ID":"26649b35-b51b-4de0-b60d-95cd8ce67d9a","Type":"ContainerStarted","Data":"232e758b6ba91ae672ef3ad42fd70a5aa6396091cd4068a450909006d4c86d78"} Aug 04 11:26:01.546122 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:01.546039 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qfz6h/must-gather-dlqzg" event={"ID":"26649b35-b51b-4de0-b60d-95cd8ce67d9a","Type":"ContainerStarted","Data":"9b71705a43eb0dfa22301b8ac97077ee5d29104c22910e0d05dc15bc9c7febf0"} Aug 04 11:26:01.546122 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:01.546092 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qfz6h/must-gather-dlqzg" event={"ID":"26649b35-b51b-4de0-b60d-95cd8ce67d9a","Type":"ContainerStarted","Data":"267b5928706ee805e7f969b3046d6024c602fff39a8313f657fea2f40e61906b"} Aug 04 11:26:01.564244 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:01.564177 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qfz6h/must-gather-dlqzg" podStartSLOduration=1.76484427 podStartE2EDuration="2.564154889s" totalImagesPullingTime="799.310619ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:25:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:26:01.562357982 +0000 UTC m=+3500.983245396" watchObservedRunningTime="2026-08-04 11:26:01.564154889 +0000 UTC m=+3500.985042301" Aug 04 11:26:02.202998 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:02.202954 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rzhxt_b0a98e70-0e23-4348-9b6d-93b4081069df/global-pull-secret-syncer/0.log" Aug 04 11:26:02.416801 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:02.416766 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tgfd7_e3137682-eadf-4ec1-b891-3e4fd7493df5/konnectivity-agent/0.log" Aug 04 11:26:02.485050 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:02.484976 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-113.ec2.internal_1a9bc83f737210ec581d9649412c17cd/haproxy/0.log" Aug 04 11:26:07.960399 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:07.960354 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-gwn2m_b3f3b35a-3c47-4410-afd5-671953692c16/manager/0.log" Aug 04 11:26:07.983914 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:07.983840 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-4npks_2893cf39-ba4a-49bc-9660-c3123ae361ba/manager/0.log" Aug 04 11:26:08.105985 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:08.105944 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5b65f94d4c-h56g4_96ffd714-48bf-432c-8ce1-112fb49eea4a/manager/0.log" Aug 04 11:26:08.127712 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:08.127681 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-jmqqj_b3b6de8b-8543-42eb-9292-5a925d23fe0f/limitador/0.log" Aug 04 11:26:08.278828 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:08.278746 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-qkfqh_1e55781b-f662-4806-8b91-fe7ab66de387/manager/0.log" Aug 04 11:26:10.133018 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:10.132984 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-85799b647-lxhxn_623b61a0-e92d-48c6-96db-e48c57399f90/metrics-server/0.log" Aug 04 11:26:10.228327 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:10.228289 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jrmjm_b29e6fab-f600-41b5-a265-d82afbedb3be/node-exporter/0.log" Aug 04 11:26:10.249511 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:10.249477 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jrmjm_b29e6fab-f600-41b5-a265-d82afbedb3be/kube-rbac-proxy/0.log" Aug 04 11:26:10.272962 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:10.271776 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jrmjm_b29e6fab-f600-41b5-a265-d82afbedb3be/init-textfile/0.log" Aug 04 11:26:12.691370 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:12.691326 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qfz6h/perf-node-gather-daemonset-nzkls"] Aug 04 11:26:12.699608 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:12.699574 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-nzkls" Aug 04 11:26:12.705274 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:12.705232 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qfz6h/perf-node-gather-daemonset-nzkls"] Aug 04 11:26:12.793205 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:12.793164 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dcfc7e9b-64ca-44ed-91f9-137a3e2d0882-podres\") pod \"perf-node-gather-daemonset-nzkls\" (UID: \"dcfc7e9b-64ca-44ed-91f9-137a3e2d0882\") " pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-nzkls" Aug 04 11:26:12.793414 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:12.793215 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xt29\" (UniqueName: \"kubernetes.io/projected/dcfc7e9b-64ca-44ed-91f9-137a3e2d0882-kube-api-access-4xt29\") pod \"perf-node-gather-daemonset-nzkls\" (UID: \"dcfc7e9b-64ca-44ed-91f9-137a3e2d0882\") " pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-nzkls" Aug 04 11:26:12.793414 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:12.793245 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dcfc7e9b-64ca-44ed-91f9-137a3e2d0882-sys\") pod \"perf-node-gather-daemonset-nzkls\" (UID: \"dcfc7e9b-64ca-44ed-91f9-137a3e2d0882\") " pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-nzkls" Aug 04 11:26:12.793414 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:12.793311 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dcfc7e9b-64ca-44ed-91f9-137a3e2d0882-proc\") pod \"perf-node-gather-daemonset-nzkls\" (UID: \"dcfc7e9b-64ca-44ed-91f9-137a3e2d0882\") " pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-nzkls" Aug 04 11:26:12.793414 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:12.793359 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dcfc7e9b-64ca-44ed-91f9-137a3e2d0882-lib-modules\") pod \"perf-node-gather-daemonset-nzkls\" (UID: \"dcfc7e9b-64ca-44ed-91f9-137a3e2d0882\") " pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-nzkls" Aug 04 11:26:12.894055 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:12.894010 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dcfc7e9b-64ca-44ed-91f9-137a3e2d0882-podres\") pod \"perf-node-gather-daemonset-nzkls\" (UID: \"dcfc7e9b-64ca-44ed-91f9-137a3e2d0882\") " pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-nzkls" Aug 04 11:26:12.894055 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:12.894050 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xt29\" (UniqueName: \"kubernetes.io/projected/dcfc7e9b-64ca-44ed-91f9-137a3e2d0882-kube-api-access-4xt29\") pod \"perf-node-gather-daemonset-nzkls\" (UID: \"dcfc7e9b-64ca-44ed-91f9-137a3e2d0882\") " pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-nzkls" Aug 04 11:26:12.894311 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:12.894185 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dcfc7e9b-64ca-44ed-91f9-137a3e2d0882-podres\") pod \"perf-node-gather-daemonset-nzkls\" (UID: \"dcfc7e9b-64ca-44ed-91f9-137a3e2d0882\") " pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-nzkls" Aug 04 11:26:12.894311 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:12.894234 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dcfc7e9b-64ca-44ed-91f9-137a3e2d0882-sys\") pod \"perf-node-gather-daemonset-nzkls\" (UID: \"dcfc7e9b-64ca-44ed-91f9-137a3e2d0882\") " pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-nzkls" Aug 04 11:26:12.894311 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:12.894276 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dcfc7e9b-64ca-44ed-91f9-137a3e2d0882-proc\") pod \"perf-node-gather-daemonset-nzkls\" (UID: \"dcfc7e9b-64ca-44ed-91f9-137a3e2d0882\") " pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-nzkls" Aug 04 11:26:12.894460 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:12.894318 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dcfc7e9b-64ca-44ed-91f9-137a3e2d0882-lib-modules\") pod \"perf-node-gather-daemonset-nzkls\" (UID: \"dcfc7e9b-64ca-44ed-91f9-137a3e2d0882\") " pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-nzkls" Aug 04 11:26:12.894460 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:12.894363 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dcfc7e9b-64ca-44ed-91f9-137a3e2d0882-sys\") pod \"perf-node-gather-daemonset-nzkls\" (UID: \"dcfc7e9b-64ca-44ed-91f9-137a3e2d0882\") " pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-nzkls" Aug 04 11:26:12.894460 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:12.894410 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dcfc7e9b-64ca-44ed-91f9-137a3e2d0882-proc\") pod \"perf-node-gather-daemonset-nzkls\" (UID: \"dcfc7e9b-64ca-44ed-91f9-137a3e2d0882\") " pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-nzkls" Aug 04 11:26:12.894609 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:12.894476 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dcfc7e9b-64ca-44ed-91f9-137a3e2d0882-lib-modules\") pod \"perf-node-gather-daemonset-nzkls\" (UID: \"dcfc7e9b-64ca-44ed-91f9-137a3e2d0882\") " pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-nzkls" Aug 04 11:26:12.905841 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:12.905808 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xt29\" (UniqueName: \"kubernetes.io/projected/dcfc7e9b-64ca-44ed-91f9-137a3e2d0882-kube-api-access-4xt29\") pod \"perf-node-gather-daemonset-nzkls\" (UID: \"dcfc7e9b-64ca-44ed-91f9-137a3e2d0882\") " pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-nzkls" Aug 04 11:26:13.026188 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:13.026103 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-nzkls" Aug 04 11:26:13.215354 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:13.215323 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qfz6h/perf-node-gather-daemonset-nzkls"] Aug 04 11:26:13.218525 ip-10-0-142-113 kubenswrapper[2580]: W0804 11:26:13.218493 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddcfc7e9b_64ca_44ed_91f9_137a3e2d0882.slice/crio-88e978a18f8ab9af05af340ca556417817464b28d2c7f9cd890e3de58f490095 WatchSource:0}: Error finding container 88e978a18f8ab9af05af340ca556417817464b28d2c7f9cd890e3de58f490095: Status 404 returned error can't find the container with id 88e978a18f8ab9af05af340ca556417817464b28d2c7f9cd890e3de58f490095 Aug 04 11:26:13.425886 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:13.425858 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-855557bfff-r99l4_5cb62afe-f3a2-4d0e-b7d6-4dcc2b77fae4/console/0.log" Aug 04 11:26:13.656452 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:13.656410 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-nzkls" event={"ID":"dcfc7e9b-64ca-44ed-91f9-137a3e2d0882","Type":"ContainerStarted","Data":"8f7e2bc0585576b93c87097cab740a34a3f809d66c07ab7c598b8c982dc1cb1d"} Aug 04 11:26:13.656687 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:13.656507 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-nzkls" event={"ID":"dcfc7e9b-64ca-44ed-91f9-137a3e2d0882","Type":"ContainerStarted","Data":"88e978a18f8ab9af05af340ca556417817464b28d2c7f9cd890e3de58f490095"} Aug 04 11:26:13.656687 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:13.656531 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-nzkls" Aug 04 11:26:13.676487 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:13.676436 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-nzkls" podStartSLOduration=1.6764201 podStartE2EDuration="1.6764201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 11:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:26:13.675517981 +0000 UTC m=+3513.096405407" watchObservedRunningTime="2026-08-04 11:26:13.6764201 +0000 UTC m=+3513.097307513" Aug 04 11:26:15.190247 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:15.190216 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qjvd4_2d4fd3c4-711b-4896-83a7-18dc02e85c7d/dns/0.log" Aug 04 11:26:15.209131 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:15.209091 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qjvd4_2d4fd3c4-711b-4896-83a7-18dc02e85c7d/kube-rbac-proxy/0.log" Aug 04 11:26:15.230632 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:15.230602 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m4sfm_59e9a581-bf1d-4594-8d5b-9dba10576132/dns-node-resolver/0.log" Aug 04 11:26:15.817301 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:15.817276 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bhq6d_53b096bc-7df7-4067-8c04-fd4d68746d12/node-ca/0.log" Aug 04 11:26:16.782848 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:16.782816 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv_7ee19c9c-fbf9-4f49-8520-0aedeee66aa6/istio-proxy/0.log" Aug 04 11:26:16.990996 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:16.990960 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bd95bfc5-zdsld_b112fb0c-d3f1-43c6-bef8-607a04229e08/kube-auth-proxy/0.log" Aug 04 11:26:17.019031 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:17.018997 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tqq7s_f93fb736-f6f2-4b03-9821-4036c36e2f72/istio-proxy/0.log" Aug 04 11:26:17.033676 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:17.033589 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tqq7s_f93fb736-f6f2-4b03-9821-4036c36e2f72/istio-proxy/1.log" Aug 04 11:26:17.908913 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:17.908878 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qbft2_6ecb558b-6b0a-42a7-a5ae-69fad1241694/serve-healthcheck-canary/0.log" Aug 04 11:26:18.503665 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:18.503628 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k4lhp_5c6f919e-0e11-4f1d-a2f0-1db97b80e53d/kube-rbac-proxy/0.log" Aug 04 11:26:18.521369 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:18.521343 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k4lhp_5c6f919e-0e11-4f1d-a2f0-1db97b80e53d/exporter/0.log" Aug 04 11:26:18.539912 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:18.539883 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k4lhp_5c6f919e-0e11-4f1d-a2f0-1db97b80e53d/extractor/0.log" Aug 04 11:26:19.673153 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:19.673126 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-nzkls" Aug 04 11:26:20.679946 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:20.679909 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-wnv4x_a4a17547-6158-4123-bc5f-39d76f32974a/manager/0.log" Aug 04 11:26:21.084443 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:21.084019 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-kl2rb_7b6a041d-9d50-4a1e-bbfa-527e5859d8cd/manager/0.log" Aug 04 11:26:21.254243 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:21.254215 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vq67x_f44b71f1-14e3-420e-b826-0705280ac08f/server/0.log" Aug 04 11:26:21.373915 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:21.373880 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-hjxw2_b36de69a-8276-4697-a3b3-194f302385d7/manager/0.log" Aug 04 11:26:21.396584 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:21.396536 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-598d67d848-6jswn_6765da31-9d18-4e54-bdc9-886576264de3/manager/0.log" Aug 04 11:26:21.412037 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:21.412009 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-598d67d848-6jswn_6765da31-9d18-4e54-bdc9-886576264de3/copy-manifests/0.log" Aug 04 11:26:23.285863 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:23.285831 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv_7ee19c9c-fbf9-4f49-8520-0aedeee66aa6/istio-proxy/0.log" Aug 04 11:26:23.464902 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:23.464873 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bd95bfc5-zdsld_b112fb0c-d3f1-43c6-bef8-607a04229e08/kube-auth-proxy/0.log" Aug 04 11:26:23.481958 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:23.481923 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tqq7s_f93fb736-f6f2-4b03-9821-4036c36e2f72/istio-proxy/0.log" Aug 04 11:26:23.496384 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:23.496350 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tqq7s_f93fb736-f6f2-4b03-9821-4036c36e2f72/istio-proxy/1.log" Aug 04 11:26:24.944076 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:24.943997 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-zppvv_7ee19c9c-fbf9-4f49-8520-0aedeee66aa6/istio-proxy/0.log" Aug 04 11:26:25.118887 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:25.118856 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bd95bfc5-zdsld_b112fb0c-d3f1-43c6-bef8-607a04229e08/kube-auth-proxy/0.log" Aug 04 11:26:25.142174 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:25.142146 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tqq7s_f93fb736-f6f2-4b03-9821-4036c36e2f72/istio-proxy/0.log" Aug 04 11:26:25.153973 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:25.153918 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tqq7s_f93fb736-f6f2-4b03-9821-4036c36e2f72/istio-proxy/1.log" Aug 04 11:26:29.491412 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:29.491373 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-49x8b_0626797c-56b0-4bac-b1a3-4a7d41a6707f/kube-multus/0.log" Aug 04 11:26:29.737571 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:29.737528 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r66c8_13819f5d-21f5-43f5-88f6-8bd915c7e86f/kube-multus-additional-cni-plugins/0.log" Aug 04 11:26:29.758481 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:29.758452 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r66c8_13819f5d-21f5-43f5-88f6-8bd915c7e86f/egress-router-binary-copy/0.log" Aug 04 11:26:29.781635 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:29.781610 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r66c8_13819f5d-21f5-43f5-88f6-8bd915c7e86f/cni-plugins/0.log" Aug 04 11:26:29.804722 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:29.804696 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r66c8_13819f5d-21f5-43f5-88f6-8bd915c7e86f/bond-cni-plugin/0.log" Aug 04 11:26:29.822146 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:29.822115 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r66c8_13819f5d-21f5-43f5-88f6-8bd915c7e86f/routeoverride-cni/0.log" Aug 04 11:26:29.841414 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:29.841373 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r66c8_13819f5d-21f5-43f5-88f6-8bd915c7e86f/whereabouts-cni-bincopy/0.log" Aug 04 11:26:29.859230 ip-10-0-142-113 kubenswrapper[2580]: I0804 11:26:29.859189 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r66c8_13819f5d-21f5-43f5-88f6-8bd915c7e86f/whereabouts-cni/0.log"