Jul 31 18:41:55.659850 ip-10-0-139-39 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 31 18:41:55.659862 ip-10-0-139-39 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 31 18:41:55.659871 ip-10-0-139-39 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 31 18:41:55.660172 ip-10-0-139-39 systemd[1]: Failed to start Kubernetes Kubelet. Jul 31 18:42:05.740491 ip-10-0-139-39 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 31 18:42:05.740506 ip-10-0-139-39 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 1021117192d947478c525180ebd1ffe9 -- Jul 31 18:43:43.247527 ip-10-0-139-39 systemd[1]: Starting Kubernetes Kubelet... Jul 31 18:43:43.657380 ip-10-0-139-39 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 31 18:43:43.657380 ip-10-0-139-39 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 31 18:43:43.657380 ip-10-0-139-39 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 31 18:43:43.657380 ip-10-0-139-39 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 31 18:43:43.657380 ip-10-0-139-39 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 31 18:43:43.658077 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.657441 2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 31 18:43:43.662271 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662255 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 18:43:43.662271 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662270 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 18:43:43.662334 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662274 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 18:43:43.662334 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662278 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 18:43:43.662334 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662281 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 18:43:43.662334 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662285 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 18:43:43.662334 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662288 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 18:43:43.662334 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662290 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 18:43:43.662334 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662293 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 18:43:43.662334 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662298 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 18:43:43.662334 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662301 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 18:43:43.662334 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662303 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 18:43:43.662334 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662306 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 18:43:43.662334 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662310 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 18:43:43.662334 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662312 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 18:43:43.662334 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662315 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 18:43:43.662334 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662318 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 18:43:43.662334 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662320 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 18:43:43.662334 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662323 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 18:43:43.662334 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662326 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 18:43:43.662334 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662328 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 18:43:43.662334 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662331 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 18:43:43.662809 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662334 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 18:43:43.662809 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662336 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 18:43:43.662809 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662339 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 18:43:43.662809 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662342 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 18:43:43.662809 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662345 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 18:43:43.662809 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662348 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 18:43:43.662809 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662350 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 18:43:43.662809 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662353 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 18:43:43.662809 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662355 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 18:43:43.662809 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662358 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 18:43:43.662809 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662360 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 18:43:43.662809 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662363 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 18:43:43.662809 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662366 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 18:43:43.662809 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662368 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 18:43:43.662809 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662371 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 18:43:43.662809 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662373 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 18:43:43.662809 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662376 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 18:43:43.662809 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662378 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 18:43:43.662809 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662381 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 18:43:43.662809 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662384 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 18:43:43.663321 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662387 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 18:43:43.663321 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662389 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 18:43:43.663321 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662392 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 18:43:43.663321 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662394 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 18:43:43.663321 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662397 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 18:43:43.663321 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662399 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 18:43:43.663321 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662402 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 18:43:43.663321 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662404 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 18:43:43.663321 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662407 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 18:43:43.663321 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662409 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 18:43:43.663321 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662414 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 18:43:43.663321 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662419 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 18:43:43.663321 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662422 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 18:43:43.663321 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662425 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 18:43:43.663321 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662428 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 18:43:43.663321 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662430 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 18:43:43.663321 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662433 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 18:43:43.663321 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662435 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 18:43:43.663321 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662438 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 18:43:43.663989 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662441 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 18:43:43.663989 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662443 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 18:43:43.663989 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662446 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 18:43:43.663989 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662448 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 18:43:43.663989 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662451 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 18:43:43.663989 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662453 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 18:43:43.663989 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662456 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 18:43:43.663989 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662459 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 18:43:43.663989 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662461 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 18:43:43.663989 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662464 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 18:43:43.663989 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662466 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 18:43:43.663989 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662469 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 18:43:43.663989 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662471 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 18:43:43.663989 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662475 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 18:43:43.663989 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662478 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 18:43:43.663989 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662483 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 18:43:43.663989 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662486 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 18:43:43.663989 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662489 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 18:43:43.663989 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662492 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 18:43:43.664603 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662495 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 31 18:43:43.664603 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662498 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 18:43:43.664603 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662502 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 18:43:43.664603 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662505 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 18:43:43.664603 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662508 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 18:43:43.664603 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662511 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 18:43:43.664603 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662513 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 18:43:43.664603 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662517 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 18:43:43.664603 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662520 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 18:43:43.664603 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662523 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 18:43:43.664603 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662526 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 18:43:43.664603 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662528 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 18:43:43.664603 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662531 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 18:43:43.664603 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662534 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 18:43:43.664603 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662537 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 18:43:43.664603 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662539 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 18:43:43.664603 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662542 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 18:43:43.664603 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662544 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 18:43:43.664603 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.662547 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 18:43:43.664603 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663565 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 18:43:43.665546 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663578 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 18:43:43.665546 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663583 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 18:43:43.665546 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663589 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 18:43:43.665546 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663594 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 18:43:43.665546 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663604 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 18:43:43.665546 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663609 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 18:43:43.665546 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663614 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 31 18:43:43.665546 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663618 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 18:43:43.665546 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663633 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 18:43:43.665546 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663638 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 18:43:43.665546 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663642 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 18:43:43.665546 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663646 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 18:43:43.665546 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663651 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 18:43:43.665546 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663655 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 18:43:43.665546 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663660 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 18:43:43.665546 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663664 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 18:43:43.665546 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663673 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 18:43:43.665546 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663677 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 18:43:43.665546 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663682 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 18:43:43.666406 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663687 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 18:43:43.666406 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663692 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 18:43:43.666406 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663696 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 18:43:43.666406 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663700 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 18:43:43.666406 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663705 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 18:43:43.666406 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663710 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 18:43:43.666406 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663723 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 18:43:43.666406 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663728 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 18:43:43.666406 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663732 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 18:43:43.666406 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663737 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 18:43:43.666406 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663745 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 18:43:43.666406 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663749 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 18:43:43.666406 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663754 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 18:43:43.666406 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663758 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 18:43:43.666406 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663766 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 18:43:43.666406 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663771 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 18:43:43.666406 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663776 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 18:43:43.666406 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663780 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 18:43:43.666406 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663784 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 18:43:43.666406 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663788 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 18:43:43.667269 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663792 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 18:43:43.667269 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663796 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 18:43:43.667269 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663815 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 18:43:43.667269 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663819 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 18:43:43.667269 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663824 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 18:43:43.667269 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663828 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 18:43:43.667269 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663832 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 18:43:43.667269 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663836 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 18:43:43.667269 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663841 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 18:43:43.667269 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663845 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 18:43:43.667269 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663849 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 18:43:43.667269 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663853 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 18:43:43.667269 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663859 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 18:43:43.667269 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663863 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 18:43:43.667269 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663872 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 18:43:43.667269 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663876 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 18:43:43.667269 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663881 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 18:43:43.667269 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663885 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 18:43:43.667269 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663890 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 18:43:43.667269 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663894 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 18:43:43.667780 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663899 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 18:43:43.667780 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663903 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 18:43:43.667780 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663907 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 18:43:43.667780 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663911 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 18:43:43.667780 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663916 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 18:43:43.667780 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663920 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 18:43:43.667780 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663924 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 18:43:43.667780 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663933 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 18:43:43.667780 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663937 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 18:43:43.667780 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663941 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 18:43:43.667780 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663946 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 18:43:43.667780 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663950 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 18:43:43.667780 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663955 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 18:43:43.667780 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663959 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 18:43:43.667780 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663963 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 18:43:43.667780 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663968 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 18:43:43.667780 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663973 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 18:43:43.667780 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663977 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 18:43:43.667780 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663982 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 18:43:43.667780 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663986 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 18:43:43.668391 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.663996 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 18:43:43.668391 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.664000 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 18:43:43.668391 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.664004 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 18:43:43.668391 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.664009 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 18:43:43.668391 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.664016 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 18:43:43.668391 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.664021 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 18:43:43.668391 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.664025 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 18:43:43.668391 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.664030 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 18:43:43.668391 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.664037 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 18:43:43.668391 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.664043 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 18:43:43.668391 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.664047 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 18:43:43.668391 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.664057 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 18:43:43.668391 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.664061 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 18:43:43.668391 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.664065 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 18:43:43.668391 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.664070 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 18:43:43.668391 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.664074 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 18:43:43.668391 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.664080 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 18:43:43.668391 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.664086 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 18:43:43.668391 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.664091 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 18:43:43.669055 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665097 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 18:43:43.669055 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665111 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 18:43:43.669055 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665116 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 18:43:43.669055 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665120 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 18:43:43.669055 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665125 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 18:43:43.669055 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665130 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 18:43:43.669055 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665134 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 18:43:43.669055 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665138 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 18:43:43.669055 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665142 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 18:43:43.669055 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665146 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 18:43:43.669055 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665150 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 18:43:43.669055 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665154 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 18:43:43.669055 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665159 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 18:43:43.669055 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665163 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 18:43:43.669055 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665167 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 18:43:43.669055 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665172 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 18:43:43.669055 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665176 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 18:43:43.669055 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665180 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 18:43:43.669055 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665183 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 18:43:43.669582 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665189 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 18:43:43.669582 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665193 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 18:43:43.669582 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665198 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 18:43:43.669582 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665202 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 18:43:43.669582 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665206 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 18:43:43.669582 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665211 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 18:43:43.669582 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665215 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 18:43:43.669582 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665222 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 18:43:43.669582 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665229 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 18:43:43.669582 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665256 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 18:43:43.669582 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665262 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 18:43:43.669582 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665266 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 18:43:43.669582 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665272 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 18:43:43.669582 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665276 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 18:43:43.669582 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665280 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 18:43:43.669582 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665285 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 18:43:43.669582 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665289 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 31 18:43:43.669582 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665293 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 18:43:43.669582 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665299 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 18:43:43.670076 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665304 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 18:43:43.670076 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665307 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 18:43:43.670076 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665312 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 18:43:43.670076 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665319 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 18:43:43.670076 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665324 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 18:43:43.670076 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665329 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 18:43:43.670076 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665334 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 18:43:43.670076 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665338 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 18:43:43.670076 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665342 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 18:43:43.670076 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665347 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 18:43:43.670076 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665351 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 18:43:43.670076 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665355 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 18:43:43.670076 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665359 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 18:43:43.670076 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665364 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 18:43:43.670076 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665368 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 18:43:43.670076 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665373 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 18:43:43.670076 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665377 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 18:43:43.670076 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665382 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 18:43:43.670076 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665386 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 18:43:43.670642 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665390 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 18:43:43.670642 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665396 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 18:43:43.670642 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665400 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 18:43:43.670642 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665405 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 18:43:43.670642 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665409 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 18:43:43.670642 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665413 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 18:43:43.670642 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665417 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 18:43:43.670642 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665421 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 18:43:43.670642 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665425 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 18:43:43.670642 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665429 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 18:43:43.670642 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665434 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 18:43:43.670642 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665438 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 18:43:43.670642 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665444 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 18:43:43.670642 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665449 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 18:43:43.670642 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665453 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 18:43:43.670642 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665457 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 18:43:43.670642 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665461 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 18:43:43.670642 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665465 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 18:43:43.670642 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665469 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 18:43:43.670642 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665473 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 18:43:43.671127 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665478 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 18:43:43.671127 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665482 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 18:43:43.671127 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665486 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 18:43:43.671127 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665490 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 18:43:43.671127 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665495 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 18:43:43.671127 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665500 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 18:43:43.671127 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665505 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 18:43:43.671127 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665509 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 18:43:43.671127 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665514 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 18:43:43.671127 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665518 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 18:43:43.671127 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665522 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 18:43:43.671127 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665526 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 18:43:43.671127 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665531 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 18:43:43.671127 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665535 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 18:43:43.671127 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665539 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 18:43:43.671127 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665543 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 18:43:43.671127 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665547 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 18:43:43.671127 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665551 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 18:43:43.671127 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665555 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 18:43:43.671127 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665559 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 18:43:43.671681 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665563 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 18:43:43.671681 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.665567 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 18:43:43.671681 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665663 2565 flags.go:64] FLAG: --address="0.0.0.0" Jul 31 18:43:43.671681 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665679 2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 31 18:43:43.671681 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665688 2565 flags.go:64] FLAG: --anonymous-auth="true" Jul 31 18:43:43.671681 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665694 2565 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 31 18:43:43.671681 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665701 2565 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 31 18:43:43.671681 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665706 2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 31 18:43:43.671681 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665716 2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 31 18:43:43.671681 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665722 2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 31 18:43:43.671681 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665727 2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 31 18:43:43.671681 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665732 2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 31 18:43:43.671681 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665737 2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 31 18:43:43.671681 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665742 2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 31 18:43:43.671681 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665747 2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 31 18:43:43.671681 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665752 2565 flags.go:64] FLAG: --cgroup-root="" Jul 31 18:43:43.671681 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665757 2565 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 31 18:43:43.671681 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665761 2565 flags.go:64] FLAG: --client-ca-file="" Jul 31 18:43:43.671681 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665767 2565 flags.go:64] FLAG: --cloud-provider="external" Jul 31 18:43:43.671681 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665772 2565 flags.go:64] FLAG: --cluster-dns="[]" Jul 31 18:43:43.671681 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665779 2565 flags.go:64] FLAG: --cluster-domain="" Jul 31 18:43:43.671681 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665784 2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 31 18:43:43.671681 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665789 2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 31 18:43:43.671681 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665794 2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 31 18:43:43.672285 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665799 2565 flags.go:64] FLAG: --container-log-max-files="5" Jul 31 18:43:43.672285 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665805 2565 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 31 18:43:43.672285 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665810 2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 31 18:43:43.672285 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665815 2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 31 18:43:43.672285 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665820 2565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 31 18:43:43.672285 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665825 2565 flags.go:64] FLAG: --contention-profiling="false" Jul 31 18:43:43.672285 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665830 2565 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 31 18:43:43.672285 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665835 2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 31 18:43:43.672285 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665841 2565 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 31 18:43:43.672285 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665845 2565 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 31 18:43:43.672285 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665852 2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 31 18:43:43.672285 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665856 2565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 31 18:43:43.672285 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665861 2565 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 31 18:43:43.672285 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665865 2565 flags.go:64] FLAG: --enable-load-reader="false" Jul 31 18:43:43.672285 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665871 2565 flags.go:64] FLAG: --enable-server="true" Jul 31 18:43:43.672285 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665875 2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 31 18:43:43.672285 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665884 2565 flags.go:64] FLAG: --event-burst="100" Jul 31 18:43:43.672285 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665889 2565 flags.go:64] FLAG: --event-qps="50" Jul 31 18:43:43.672285 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665894 2565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 31 18:43:43.672285 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665898 2565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 31 18:43:43.672285 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665903 2565 flags.go:64] FLAG: --eviction-hard="" Jul 31 18:43:43.672285 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665909 2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 31 18:43:43.672285 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665914 2565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 31 18:43:43.672285 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665919 2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 31 18:43:43.672285 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665923 2565 flags.go:64] FLAG: --eviction-soft="" Jul 31 18:43:43.672900 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665928 2565 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 31 18:43:43.672900 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665933 2565 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 31 18:43:43.672900 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665938 2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 31 18:43:43.672900 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665943 2565 flags.go:64] FLAG: --experimental-mounter-path="" Jul 31 18:43:43.672900 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665948 2565 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 31 18:43:43.672900 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665953 2565 flags.go:64] FLAG: --fail-swap-on="true" Jul 31 18:43:43.672900 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665957 2565 flags.go:64] FLAG: --feature-gates="" Jul 31 18:43:43.672900 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665963 2565 flags.go:64] FLAG: --file-check-frequency="20s" Jul 31 18:43:43.672900 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665968 2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 31 18:43:43.672900 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665973 2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 31 18:43:43.672900 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665977 2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 31 18:43:43.672900 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665982 2565 flags.go:64] FLAG: --healthz-port="10248" Jul 31 18:43:43.672900 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665987 2565 flags.go:64] FLAG: --help="false" Jul 31 18:43:43.672900 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665992 2565 flags.go:64] FLAG: --hostname-override="ip-10-0-139-39.ec2.internal" Jul 31 18:43:43.672900 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.665997 2565 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 31 18:43:43.672900 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666002 2565 flags.go:64] FLAG: --http-check-frequency="20s" Jul 31 18:43:43.672900 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666007 2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 31 18:43:43.672900 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666012 2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 31 18:43:43.672900 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666018 2565 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 31 18:43:43.672900 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666022 2565 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 31 18:43:43.672900 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666027 2565 flags.go:64] FLAG: --image-service-endpoint="" Jul 31 18:43:43.672900 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666032 2565 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 31 18:43:43.672900 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666037 2565 flags.go:64] FLAG: --kube-api-burst="100" Jul 31 18:43:43.673524 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666044 2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 31 18:43:43.673524 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666050 2565 flags.go:64] FLAG: --kube-api-qps="50" Jul 31 18:43:43.673524 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666054 2565 flags.go:64] FLAG: --kube-reserved="" Jul 31 18:43:43.673524 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666059 2565 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 31 18:43:43.673524 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666063 2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 31 18:43:43.673524 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666068 2565 flags.go:64] FLAG: --kubelet-cgroups="" Jul 31 18:43:43.673524 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666073 2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 31 18:43:43.673524 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666077 2565 flags.go:64] FLAG: --lock-file="" Jul 31 18:43:43.673524 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666082 2565 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 31 18:43:43.673524 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666086 2565 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 31 18:43:43.673524 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666092 2565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 31 18:43:43.673524 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666101 2565 flags.go:64] FLAG: --log-json-split-stream="false" Jul 31 18:43:43.673524 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666107 2565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 31 18:43:43.673524 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666111 2565 flags.go:64] FLAG: --log-text-split-stream="false" Jul 31 18:43:43.673524 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666116 2565 flags.go:64] FLAG: --logging-format="text" Jul 31 18:43:43.673524 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666120 2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 31 18:43:43.673524 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666126 2565 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 31 18:43:43.673524 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666131 2565 flags.go:64] FLAG: --manifest-url="" Jul 31 18:43:43.673524 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666136 2565 flags.go:64] FLAG: --manifest-url-header="" Jul 31 18:43:43.673524 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666150 2565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 31 18:43:43.673524 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666155 2565 flags.go:64] FLAG: --max-open-files="1000000" Jul 31 18:43:43.673524 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666161 2565 flags.go:64] FLAG: --max-pods="110" Jul 31 18:43:43.673524 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666166 2565 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 31 18:43:43.673524 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666170 2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 31 18:43:43.673524 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666175 2565 flags.go:64] FLAG: --memory-manager-policy="None" Jul 31 18:43:43.674122 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666180 2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 31 18:43:43.674122 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666185 2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 31 18:43:43.674122 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666190 2565 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 31 18:43:43.674122 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666194 2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 31 18:43:43.674122 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666206 2565 flags.go:64] FLAG: --node-status-max-images="50" Jul 31 18:43:43.674122 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666211 2565 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 31 18:43:43.674122 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666216 2565 flags.go:64] FLAG: --oom-score-adj="-999" Jul 31 18:43:43.674122 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666223 2565 flags.go:64] FLAG: --pod-cidr="" Jul 31 18:43:43.674122 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666227 2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Jul 31 18:43:43.674122 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666253 2565 flags.go:64] FLAG: --pod-manifest-path="" Jul 31 18:43:43.674122 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666258 2565 flags.go:64] FLAG: --pod-max-pids="-1" Jul 31 18:43:43.674122 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666264 2565 flags.go:64] FLAG: --pods-per-core="0" Jul 31 18:43:43.674122 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666268 2565 flags.go:64] FLAG: --port="10250" Jul 31 18:43:43.674122 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666273 2565 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 31 18:43:43.674122 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666278 2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-098c5d4fecd2353ea" Jul 31 18:43:43.674122 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666283 2565 flags.go:64] FLAG: --qos-reserved="" Jul 31 18:43:43.674122 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666287 2565 flags.go:64] FLAG: --read-only-port="10255" Jul 31 18:43:43.674122 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666293 2565 flags.go:64] FLAG: --register-node="true" Jul 31 18:43:43.674122 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666300 2565 flags.go:64] FLAG: --register-with-taints="" Jul 31 18:43:43.674122 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666306 2565 flags.go:64] FLAG: --registry-burst="10" Jul 31 18:43:43.674122 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666312 2565 flags.go:64] FLAG: --registry-qps="5" Jul 31 18:43:43.674122 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666317 2565 flags.go:64] FLAG: --reserved-cpus="" Jul 31 18:43:43.674122 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666321 2565 flags.go:64] FLAG: --reserved-memory="" Jul 31 18:43:43.674122 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666327 2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 31 18:43:43.674122 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666332 2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 31 18:43:43.674758 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666337 2565 flags.go:64] FLAG: --rotate-certificates="false" Jul 31 18:43:43.674758 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666342 2565 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 31 18:43:43.674758 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666347 2565 flags.go:64] FLAG: --runonce="false" Jul 31 18:43:43.674758 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666352 2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 31 18:43:43.674758 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666357 2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 31 18:43:43.674758 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666362 2565 flags.go:64] FLAG: --seccomp-default="false" Jul 31 18:43:43.674758 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666366 2565 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 31 18:43:43.674758 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666371 2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 31 18:43:43.674758 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666376 2565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 31 18:43:43.674758 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666381 2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 31 18:43:43.674758 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666386 2565 flags.go:64] FLAG: --storage-driver-password="root" Jul 31 18:43:43.674758 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666390 2565 flags.go:64] FLAG: --storage-driver-secure="false" Jul 31 18:43:43.674758 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666395 2565 flags.go:64] FLAG: --storage-driver-table="stats" Jul 31 18:43:43.674758 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666400 2565 flags.go:64] FLAG: --storage-driver-user="root" Jul 31 18:43:43.674758 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666406 2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 31 18:43:43.674758 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666411 2565 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 31 18:43:43.674758 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666415 2565 flags.go:64] FLAG: --system-cgroups="" Jul 31 18:43:43.674758 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666420 2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 31 18:43:43.674758 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666428 2565 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 31 18:43:43.674758 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666433 2565 flags.go:64] FLAG: --tls-cert-file="" Jul 31 18:43:43.674758 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666438 2565 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 31 18:43:43.674758 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666444 2565 flags.go:64] FLAG: --tls-min-version="" Jul 31 18:43:43.674758 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666449 2565 flags.go:64] FLAG: --tls-private-key-file="" Jul 31 18:43:43.674758 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666454 2565 flags.go:64] FLAG: --topology-manager-policy="none" Jul 31 18:43:43.674758 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666458 2565 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 31 18:43:43.675404 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666468 2565 flags.go:64] FLAG: --topology-manager-scope="container" Jul 31 18:43:43.675404 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666472 2565 flags.go:64] FLAG: --v="2" Jul 31 18:43:43.675404 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666479 2565 flags.go:64] FLAG: --version="false" Jul 31 18:43:43.675404 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666486 2565 flags.go:64] FLAG: --vmodule="" Jul 31 18:43:43.675404 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666492 2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 31 18:43:43.675404 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.666498 2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 31 18:43:43.675404 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666677 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 18:43:43.675404 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666683 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 18:43:43.675404 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666689 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 18:43:43.675404 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666693 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 18:43:43.675404 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666700 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 18:43:43.675404 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666706 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 18:43:43.675404 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666711 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 18:43:43.675404 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666715 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 18:43:43.675404 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666723 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 18:43:43.675404 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666729 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 18:43:43.675404 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666734 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 18:43:43.675404 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666739 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 18:43:43.675404 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666744 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 18:43:43.675404 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666749 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 18:43:43.675893 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666754 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 18:43:43.675893 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666760 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 18:43:43.675893 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666764 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 18:43:43.675893 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666769 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 18:43:43.675893 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666774 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 18:43:43.675893 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666779 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 18:43:43.675893 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666784 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 18:43:43.675893 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666789 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 18:43:43.675893 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666793 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 18:43:43.675893 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666798 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 18:43:43.675893 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666803 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 18:43:43.675893 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666807 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 18:43:43.675893 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666813 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 31 18:43:43.675893 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666818 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 18:43:43.675893 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666822 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 18:43:43.675893 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666826 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 18:43:43.675893 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666831 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 18:43:43.675893 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666869 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 18:43:43.675893 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666876 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 18:43:43.675893 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666880 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 18:43:43.676411 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666885 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 18:43:43.676411 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666889 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 18:43:43.676411 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666893 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 18:43:43.676411 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666897 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 18:43:43.676411 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666902 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 18:43:43.676411 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666906 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 18:43:43.676411 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666911 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 18:43:43.676411 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666915 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 18:43:43.676411 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666919 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 18:43:43.676411 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666924 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 18:43:43.676411 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666928 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 18:43:43.676411 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666932 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 18:43:43.676411 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666936 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 18:43:43.676411 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666943 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 18:43:43.676411 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666948 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 18:43:43.676411 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666952 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 18:43:43.676411 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666956 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 18:43:43.676411 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666960 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 18:43:43.676411 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666964 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 18:43:43.676411 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666968 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 18:43:43.676885 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666972 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 18:43:43.676885 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666977 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 18:43:43.676885 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666982 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 18:43:43.676885 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666986 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 18:43:43.676885 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666992 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 18:43:43.676885 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.666996 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 18:43:43.676885 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667001 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 18:43:43.676885 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667005 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 18:43:43.676885 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667009 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 18:43:43.676885 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667014 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 18:43:43.676885 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667018 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 18:43:43.676885 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667023 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 18:43:43.676885 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667027 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 18:43:43.676885 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667031 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 18:43:43.676885 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667036 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 18:43:43.676885 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667040 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 18:43:43.676885 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667044 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 18:43:43.676885 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667048 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 18:43:43.676885 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667052 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 18:43:43.676885 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667057 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 18:43:43.677379 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667061 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 18:43:43.677379 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667065 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 18:43:43.677379 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667069 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 18:43:43.677379 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667073 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 18:43:43.677379 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667078 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 18:43:43.677379 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667084 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 18:43:43.677379 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667088 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 18:43:43.677379 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667092 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 18:43:43.677379 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667096 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 18:43:43.677379 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667100 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 18:43:43.677379 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667104 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 18:43:43.677379 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667108 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 18:43:43.677379 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667112 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 18:43:43.677379 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667118 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 18:43:43.677379 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667125 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 18:43:43.677379 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667130 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 18:43:43.677379 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667136 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 18:43:43.677379 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667140 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 18:43:43.677379 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667144 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 18:43:43.677379 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667148 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 18:43:43.677877 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667152 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 18:43:43.677877 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667157 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 18:43:43.677877 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667162 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 18:43:43.677877 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667166 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 18:43:43.677877 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.667170 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 18:43:43.677877 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.667787 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 31 18:43:43.677877 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.675198 2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 31 18:43:43.677877 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.675213 2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 31 18:43:43.677877 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.675229 2565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 31 18:43:43.677877 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.675251 2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 31 18:43:43.677877 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675312 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 18:43:43.677877 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675318 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 18:43:43.677877 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675321 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 18:43:43.677877 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675325 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 18:43:43.678220 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675330 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 18:43:43.678220 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675333 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 18:43:43.678220 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675337 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 18:43:43.678220 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675340 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 18:43:43.678220 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675343 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 18:43:43.678220 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675346 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 18:43:43.678220 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675349 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 18:43:43.678220 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675352 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 18:43:43.678220 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675355 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 18:43:43.678220 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675358 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 18:43:43.678220 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675360 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 18:43:43.678220 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675363 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 18:43:43.678220 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675365 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 18:43:43.678220 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675381 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 18:43:43.678220 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675386 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 18:43:43.678220 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675389 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 18:43:43.678220 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675392 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 18:43:43.678220 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675395 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 18:43:43.678220 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675398 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 18:43:43.678756 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675401 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 18:43:43.678756 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675404 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 18:43:43.678756 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675408 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 18:43:43.678756 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675411 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 18:43:43.678756 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675414 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 18:43:43.678756 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675417 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 18:43:43.678756 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675421 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 18:43:43.678756 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675424 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 18:43:43.678756 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675427 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 18:43:43.678756 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675429 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 18:43:43.678756 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675432 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 18:43:43.678756 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675435 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 18:43:43.678756 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675437 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 18:43:43.678756 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675440 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 18:43:43.678756 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675442 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 18:43:43.678756 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675445 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 18:43:43.678756 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675448 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 18:43:43.678756 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675450 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 18:43:43.678756 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675453 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 18:43:43.679211 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675456 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 18:43:43.679211 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675458 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 18:43:43.679211 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675461 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 18:43:43.679211 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675463 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 18:43:43.679211 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675466 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 18:43:43.679211 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675469 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 18:43:43.679211 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675471 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 18:43:43.679211 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675474 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 18:43:43.679211 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675476 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 18:43:43.679211 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675480 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 18:43:43.679211 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675482 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 18:43:43.679211 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675485 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 18:43:43.679211 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675487 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 18:43:43.679211 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675490 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 18:43:43.679211 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675492 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 18:43:43.679211 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675495 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 18:43:43.679211 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675498 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 18:43:43.679211 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675501 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 18:43:43.679211 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675503 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 18:43:43.679211 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675506 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 31 18:43:43.679726 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675510 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 18:43:43.679726 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675512 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 18:43:43.679726 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675515 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 18:43:43.679726 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675517 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 18:43:43.679726 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675521 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 18:43:43.679726 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675524 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 18:43:43.679726 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675527 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 18:43:43.679726 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675529 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 18:43:43.679726 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675532 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 18:43:43.679726 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675535 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 18:43:43.679726 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675537 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 18:43:43.679726 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675540 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 18:43:43.679726 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675542 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 18:43:43.679726 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675545 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 18:43:43.679726 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675547 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 18:43:43.679726 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675550 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 18:43:43.679726 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675552 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 18:43:43.679726 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675555 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 18:43:43.679726 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675558 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 18:43:43.680194 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675560 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 18:43:43.680194 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675563 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 18:43:43.680194 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675565 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 18:43:43.680194 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675568 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 18:43:43.680194 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675570 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 18:43:43.680194 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675573 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 18:43:43.680194 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675575 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 18:43:43.680194 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675578 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 18:43:43.680194 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675580 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 18:43:43.680194 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675583 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 18:43:43.680194 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675586 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 18:43:43.680194 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675588 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 18:43:43.680194 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675591 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 18:43:43.680194 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675594 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 18:43:43.680194 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675597 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 18:43:43.680194 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675600 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 18:43:43.680194 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675602 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 18:43:43.680194 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675605 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 18:43:43.680644 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.675609 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 31 18:43:43.680644 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675751 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 18:43:43.680644 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675756 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 18:43:43.680644 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675760 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 18:43:43.680644 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675763 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 18:43:43.680644 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675765 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 18:43:43.680644 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675768 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 18:43:43.680644 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675771 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 18:43:43.680644 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675773 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 18:43:43.680644 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675776 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 18:43:43.680644 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675778 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 18:43:43.680644 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675781 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 18:43:43.680644 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675783 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 18:43:43.680644 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675786 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 18:43:43.680644 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675788 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 18:43:43.681015 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675791 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 18:43:43.681015 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675793 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 18:43:43.681015 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675796 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 18:43:43.681015 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675799 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 18:43:43.681015 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675802 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 18:43:43.681015 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675804 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 18:43:43.681015 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675807 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 18:43:43.681015 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675809 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 18:43:43.681015 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675812 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 18:43:43.681015 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675816 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 18:43:43.681015 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675819 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 18:43:43.681015 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675822 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 18:43:43.681015 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675825 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 18:43:43.681015 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675827 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 18:43:43.681015 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675831 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 18:43:43.681015 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675833 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 18:43:43.681015 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675836 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 18:43:43.681015 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675838 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 18:43:43.681015 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675841 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 18:43:43.681492 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675844 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 18:43:43.681492 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675847 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 18:43:43.681492 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675849 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 18:43:43.681492 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675853 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 18:43:43.681492 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675856 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 18:43:43.681492 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675860 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 18:43:43.681492 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675862 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 18:43:43.681492 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675865 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 18:43:43.681492 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675868 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 18:43:43.681492 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675871 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 18:43:43.681492 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675873 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 18:43:43.681492 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675876 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 18:43:43.681492 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675879 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 18:43:43.681492 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675881 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 18:43:43.681492 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675884 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 18:43:43.681492 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675886 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 18:43:43.681492 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675891 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 18:43:43.681492 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675895 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 18:43:43.681492 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675898 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 18:43:43.681943 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675901 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 18:43:43.681943 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675904 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 18:43:43.681943 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675907 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 18:43:43.681943 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675910 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 18:43:43.681943 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675912 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 18:43:43.681943 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675915 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 18:43:43.681943 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675917 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 18:43:43.681943 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675920 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 18:43:43.681943 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675922 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 18:43:43.681943 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675926 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 18:43:43.681943 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675928 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 18:43:43.681943 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675931 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 18:43:43.681943 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675934 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 18:43:43.681943 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675936 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 18:43:43.681943 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675939 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 18:43:43.681943 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675941 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 18:43:43.681943 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675944 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 18:43:43.681943 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675947 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 18:43:43.681943 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675949 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 18:43:43.681943 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675952 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 18:43:43.682457 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675954 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 18:43:43.682457 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675957 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 18:43:43.682457 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675960 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 18:43:43.682457 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675962 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 18:43:43.682457 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675965 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 18:43:43.682457 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675968 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 31 18:43:43.682457 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675970 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 18:43:43.682457 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675973 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 18:43:43.682457 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675975 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 18:43:43.682457 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675978 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 18:43:43.682457 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675980 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 18:43:43.682457 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675982 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 18:43:43.682457 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675985 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 18:43:43.682457 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675987 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 18:43:43.682457 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675990 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 18:43:43.682457 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675992 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 18:43:43.682457 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675995 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 18:43:43.682457 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.675998 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 18:43:43.682457 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.676001 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 18:43:43.682457 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.676003 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 18:43:43.682934 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.676005 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 18:43:43.682934 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.676008 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 18:43:43.682934 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.676011 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 18:43:43.682934 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.676013 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 18:43:43.682934 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.676016 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 18:43:43.682934 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.676018 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 18:43:43.682934 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:43.676021 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 18:43:43.682934 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.676026 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 31 18:43:43.682934 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.676134 2565 server.go:962] "Client rotation is on, will bootstrap in background" Jul 31 18:43:43.682934 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.678589 2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 31 18:43:43.682934 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.680153 2565 server.go:1019] "Starting client certificate rotation" Jul 31 18:43:43.682934 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.680258 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 31 18:43:43.682934 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.680329 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 31 18:43:43.683370 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.683059 2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 31 18:43:43.684290 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.684260 2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 31 18:43:43.701837 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.701815 2565 log.go:25] "Validated CRI v1 runtime API" Jul 31 18:43:43.704031 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.704017 2565 log.go:25] "Validated CRI v1 image API" Jul 31 18:43:43.704785 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.704765 2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 31 18:43:43.707291 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.707269 2565 fs.go:135] Filesystem UUIDs: map[2895c78e-8fdb-4210-94ea-c88885a131bf:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f13012a3-1ab9-4a47-8d85-a180dfbc4693:/dev/nvme0n1p4] Jul 31 18:43:43.707381 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.707289 2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 31 18:43:43.712491 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.712373 2565 manager.go:216] Machine: {Timestamp:2026-07-31 18:43:43.710948535 +0000 UTC m=+0.348938325 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100077 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c072a9257b84dd63440eec1867392 SystemUUID:ec2c072a-9257-b84d-d634-40eec1867392 BootID:10211171-92d9-4747-8c52-5180ebd1ffe9 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:08:9e:73:bd:df Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:08:9e:73:bd:df Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:10:40:42:fa:bf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 31 18:43:43.712491 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.712481 2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 31 18:43:43.712647 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.712587 2565 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 31 18:43:43.713006 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.712989 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 31 18:43:43.716567 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.716544 2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 31 18:43:43.716746 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.716570 2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-39.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 31 18:43:43.717708 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.717696 2565 topology_manager.go:138] "Creating topology manager with none policy" Jul 31 18:43:43.717771 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.717716 2565 container_manager_linux.go:309] "Creating device plugin manager" Jul 31 18:43:43.717771 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.717728 2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 31 18:43:43.718673 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.718661 2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 31 18:43:43.719660 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.719649 2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 31 18:43:43.723360 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.723346 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 31 18:43:43.723494 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.723483 2565 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 31 18:43:43.723550 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.723523 2565 kubelet.go:486] "Attempting to sync node with API server" Jul 31 18:43:43.723550 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.723541 2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 31 18:43:43.723642 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.723560 2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 31 18:43:43.723642 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.723574 2565 kubelet.go:398] "Adding apiserver pod source" Jul 31 18:43:43.723642 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.723596 2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 31 18:43:43.723966 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.723951 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 31 18:43:43.724031 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.723975 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 31 18:43:43.726531 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.726513 2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 31 18:43:43.728262 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.728233 2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 31 18:43:43.728305 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.728278 2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 31 18:43:43.729582 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.729571 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 31 18:43:43.729625 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.729588 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 31 18:43:43.729625 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.729594 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 31 18:43:43.729625 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.729599 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 31 18:43:43.729625 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.729610 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 31 18:43:43.729625 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.729618 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 31 18:43:43.729817 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.729629 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 31 18:43:43.729817 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.729635 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 31 18:43:43.729817 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.729642 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 31 18:43:43.729817 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.729647 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 31 18:43:43.729817 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.729658 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 31 18:43:43.729817 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.729666 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 31 18:43:43.731255 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.731232 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 31 18:43:43.731292 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.731258 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 31 18:43:43.732873 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.732841 2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-39.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 31 18:43:43.733420 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:43.733390 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-39.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 31 18:43:43.733469 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:43.733390 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 31 18:43:43.734529 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.734516 2565 server.go:1268] "Started kubelet" Jul 31 18:43:43.734675 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.734646 2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 31 18:43:43.734675 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.734639 2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 31 18:43:43.734817 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.734740 2565 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 31 18:43:43.735471 ip-10-0-139-39 systemd[1]: Started Kubernetes Kubelet. Jul 31 18:43:43.740907 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.740883 2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 31 18:43:43.740988 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.740823 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fqlrs" Jul 31 18:43:43.742515 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.742498 2565 server.go:310] "Adding debug handlers to kubelet server" Jul 31 18:43:43.743812 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:43.743776 2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 31 18:43:43.745181 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:43.744181 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-39.ec2.internal.18c7741801d57db4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-39.ec2.internal,UID:ip-10-0-139-39.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-39.ec2.internal,},FirstTimestamp:2026-07-31 18:43:43.734496692 +0000 UTC m=+0.372486483,LastTimestamp:2026-07-31 18:43:43.734496692 +0000 UTC m=+0.372486483,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-39.ec2.internal,}" Jul 31 18:43:43.747128 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.747114 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fqlrs" Jul 31 18:43:43.747474 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.747458 2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 31 18:43:43.747560 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.747541 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 31 18:43:43.747631 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:43.747610 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-39.ec2.internal\" not found" Jul 31 18:43:43.748274 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.748256 2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 31 18:43:43.748274 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.748257 2565 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 31 18:43:43.748444 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.748282 2565 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 31 18:43:43.748500 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.748467 2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 31 18:43:43.748500 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.748480 2565 factory.go:55] Registering systemd factory Jul 31 18:43:43.748500 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.748486 2565 factory.go:223] Registration of the systemd container factory successfully Jul 31 18:43:43.748637 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.748485 2565 reconstruct.go:97] "Volume reconstruction finished" Jul 31 18:43:43.748637 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.748556 2565 reconciler.go:29] "Reconciler: start to sync state" Jul 31 18:43:43.748721 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.748707 2565 factory.go:153] Registering CRI-O factory Jul 31 18:43:43.748721 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.748719 2565 factory.go:223] Registration of the crio container factory successfully Jul 31 18:43:43.748798 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.748739 2565 factory.go:103] Registering Raw factory Jul 31 18:43:43.748798 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.748750 2565 manager.go:1196] Started watching for new ooms in manager Jul 31 18:43:43.749041 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.749028 2565 manager.go:349] Starting recovery of all containers Jul 31 18:43:43.761024 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.761006 2565 manager.go:354] Recovery completed Jul 31 18:43:43.761819 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.761801 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 18:43:43.766983 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.766967 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 18:43:43.767398 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:43.767382 2565 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-39.ec2.internal\" not found" node="ip-10-0-139-39.ec2.internal" Jul 31 18:43:43.768697 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.768682 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasSufficientMemory" Jul 31 18:43:43.768816 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.768717 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 18:43:43.768816 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.768730 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasSufficientPID" Jul 31 18:43:43.769316 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.769301 2565 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 31 18:43:43.769394 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.769317 2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 31 18:43:43.769394 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.769323 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 18:43:43.769394 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.769337 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 31 18:43:43.770116 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.770103 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasSufficientMemory" Jul 31 18:43:43.770166 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.770130 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 18:43:43.770166 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.770141 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasSufficientPID" Jul 31 18:43:43.770576 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.770564 2565 policy_none.go:49] "None policy: Start" Jul 31 18:43:43.770621 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.770580 2565 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 31 18:43:43.770621 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.770590 2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 31 18:43:43.771503 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.771491 2565 policy_none.go:47] "Start" Jul 31 18:43:43.800612 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.800596 2565 manager.go:337] "Starting Device Plugin manager" Jul 31 18:43:43.817906 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:43.800635 2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 31 18:43:43.817906 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.800645 2565 server.go:87] "Starting device plugin registration server" Jul 31 18:43:43.817906 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.800887 2565 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 31 18:43:43.817906 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.800900 2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 31 18:43:43.817906 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.800984 2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 31 18:43:43.817906 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.801062 2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 31 18:43:43.817906 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.801071 2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 31 18:43:43.817906 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:43.801779 2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 31 18:43:43.817906 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:43.801834 2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-39.ec2.internal\" not found" Jul 31 18:43:43.878641 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.878615 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 31 18:43:43.879832 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.879818 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 31 18:43:43.879895 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.879856 2565 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 31 18:43:43.879895 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.879878 2565 kubelet.go:2448] "Starting kubelet main sync loop" Jul 31 18:43:43.879986 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:43.879916 2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 31 18:43:43.882718 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.882701 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 18:43:43.901908 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.901891 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 18:43:43.902797 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.902780 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasSufficientMemory" Jul 31 18:43:43.902886 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.902808 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 18:43:43.902886 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.902820 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasSufficientPID" Jul 31 18:43:43.902886 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.902842 2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-39.ec2.internal" Jul 31 18:43:43.914275 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.914208 2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-39.ec2.internal" Jul 31 18:43:43.914275 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:43.914229 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-39.ec2.internal\": node \"ip-10-0-139-39.ec2.internal\" not found" Jul 31 18:43:43.934693 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:43.934667 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-39.ec2.internal\" not found" Jul 31 18:43:43.980315 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.980278 2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-39.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal"] Jul 31 18:43:43.980418 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.980357 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 18:43:43.982098 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.982080 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasSufficientMemory" Jul 31 18:43:43.982202 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.982120 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 18:43:43.982202 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.982136 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasSufficientPID" Jul 31 18:43:43.983428 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.983413 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 18:43:43.983542 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.983526 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-39.ec2.internal" Jul 31 18:43:43.983600 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.983561 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 18:43:43.984110 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.984094 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasSufficientMemory" Jul 31 18:43:43.984189 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.984099 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasSufficientMemory" Jul 31 18:43:43.984189 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.984124 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 18:43:43.984189 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.984138 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasSufficientPID" Jul 31 18:43:43.984189 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.984143 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 18:43:43.984189 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.984154 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasSufficientPID" Jul 31 18:43:43.985189 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.985173 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal" Jul 31 18:43:43.985287 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.985217 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 18:43:43.985907 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.985895 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasSufficientMemory" Jul 31 18:43:43.985975 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.985917 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 18:43:43.985975 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:43.985927 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeHasSufficientPID" Jul 31 18:43:44.012891 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:44.012870 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-39.ec2.internal\" not found" node="ip-10-0-139-39.ec2.internal" Jul 31 18:43:44.017204 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:44.017189 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-39.ec2.internal\" not found" node="ip-10-0-139-39.ec2.internal" Jul 31 18:43:44.035421 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:44.035401 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-39.ec2.internal\" not found" Jul 31 18:43:44.051113 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:44.050957 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6781281007e7ddfaa5111df980719fef-config\") pod \"kube-apiserver-proxy-ip-10-0-139-39.ec2.internal\" (UID: \"6781281007e7ddfaa5111df980719fef\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-39.ec2.internal" Jul 31 18:43:44.051210 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:44.051122 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6277a79a23fe911d619a284a778ea843-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal\" (UID: \"6277a79a23fe911d619a284a778ea843\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal" Jul 31 18:43:44.051210 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:44.051138 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6277a79a23fe911d619a284a778ea843-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal\" (UID: \"6277a79a23fe911d619a284a778ea843\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal" Jul 31 18:43:44.135848 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:44.135811 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-39.ec2.internal\" not found" Jul 31 18:43:44.152226 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:44.152200 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6277a79a23fe911d619a284a778ea843-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal\" (UID: \"6277a79a23fe911d619a284a778ea843\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal" Jul 31 18:43:44.152325 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:44.152231 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6277a79a23fe911d619a284a778ea843-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal\" (UID: \"6277a79a23fe911d619a284a778ea843\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal" Jul 31 18:43:44.152325 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:44.152272 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6781281007e7ddfaa5111df980719fef-config\") pod \"kube-apiserver-proxy-ip-10-0-139-39.ec2.internal\" (UID: \"6781281007e7ddfaa5111df980719fef\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-39.ec2.internal" Jul 31 18:43:44.152325 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:44.152313 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6277a79a23fe911d619a284a778ea843-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal\" (UID: \"6277a79a23fe911d619a284a778ea843\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal" Jul 31 18:43:44.152421 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:44.152339 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6277a79a23fe911d619a284a778ea843-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal\" (UID: \"6277a79a23fe911d619a284a778ea843\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal" Jul 31 18:43:44.152421 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:44.152394 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6781281007e7ddfaa5111df980719fef-config\") pod \"kube-apiserver-proxy-ip-10-0-139-39.ec2.internal\" (UID: \"6781281007e7ddfaa5111df980719fef\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-39.ec2.internal" Jul 31 18:43:44.236700 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:44.236625 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-39.ec2.internal\" not found" Jul 31 18:43:44.316233 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:44.316206 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-39.ec2.internal" Jul 31 18:43:44.320222 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:44.320195 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal" Jul 31 18:43:44.336742 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:44.336720 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-39.ec2.internal\" not found" Jul 31 18:43:44.437288 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:44.437257 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-39.ec2.internal\" not found" Jul 31 18:43:44.537699 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:44.537677 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-39.ec2.internal\" not found" Jul 31 18:43:44.643825 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:44.643800 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-39.ec2.internal\" not found" Jul 31 18:43:44.679294 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:44.679272 2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 31 18:43:44.679687 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:44.679408 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 31 18:43:44.679687 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:44.679437 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 31 18:43:44.744095 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:44.744065 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-39.ec2.internal\" not found" Jul 31 18:43:44.752158 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:44.752132 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-30 18:38:44 +0000 UTC" deadline="2028-02-12 11:29:32.130140316 +0000 UTC" Jul 31 18:43:44.752158 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:44.752156 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13456h45m47.377987296s" Jul 31 18:43:44.845002 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:44.844937 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-39.ec2.internal\" not found" Jul 31 18:43:44.880066 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:44.880029 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6781281007e7ddfaa5111df980719fef.slice/crio-71c83eef61737a952e0d1568897677ca143013ad1de8891edf55ddc92c9b6d59 WatchSource:0}: Error finding container 71c83eef61737a952e0d1568897677ca143013ad1de8891edf55ddc92c9b6d59: Status 404 returned error can't find the container with id 71c83eef61737a952e0d1568897677ca143013ad1de8891edf55ddc92c9b6d59 Jul 31 18:43:44.884259 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:44.884243 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 18:43:44.948301 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:44.948270 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-39.ec2.internal\" not found" Jul 31 18:43:44.950811 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:44.950795 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 18:43:45.035570 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.035546 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 18:43:45.048552 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.048531 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-39.ec2.internal" Jul 31 18:43:45.054108 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:45.054084 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6277a79a23fe911d619a284a778ea843.slice/crio-573452a1ca41c4d0d4f287397555223254e0a8b0a4174e6b7ed58da4fc261bc4 WatchSource:0}: Error finding container 573452a1ca41c4d0d4f287397555223254e0a8b0a4174e6b7ed58da4fc261bc4: Status 404 returned error can't find the container with id 573452a1ca41c4d0d4f287397555223254e0a8b0a4174e6b7ed58da4fc261bc4 Jul 31 18:43:45.059536 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.059514 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 31 18:43:45.060626 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.060611 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal" Jul 31 18:43:45.071810 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.071793 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 31 18:43:45.532466 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.532419 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 18:43:45.724640 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.724608 2565 apiserver.go:52] "Watching apiserver" Jul 31 18:43:45.731069 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.731038 2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 31 18:43:45.731669 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.731618 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-39.ec2.internal","openshift-cluster-node-tuning-operator/tuned-jh7cb","openshift-multus/network-metrics-daemon-h8n5q","openshift-network-diagnostics/network-check-target-xqlm2","openshift-network-operator/iptables-alerter-dkbrb","openshift-ovn-kubernetes/ovnkube-node-fxm9n","kube-system/global-pull-secret-syncer-2rkhw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf2rc","openshift-dns/node-resolver-fjbvh","openshift-image-registry/node-ca-sbmpv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal","openshift-multus/multus-additional-cni-plugins-p9mqb","openshift-multus/multus-vbbdc","kube-system/konnectivity-agent-7s5zf"] Jul 31 18:43:45.734225 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.734198 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:43:45.734359 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:45.734327 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rkhw" podUID="e8223854-5549-4701-b1ce-506ed19d8abe" Jul 31 18:43:45.736746 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.736643 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.739063 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.739038 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:43:45.739063 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.739050 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 31 18:43:45.739220 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.739087 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mldd6\"" Jul 31 18:43:45.739220 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:45.739118 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8n5q" podUID="ee8aa68f-72ff-4118-ad38-4da03b305feb" Jul 31 18:43:45.739220 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.739210 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 31 18:43:45.739410 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.739290 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 31 18:43:45.739410 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.739302 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 31 18:43:45.741411 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.741389 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqlm2" Jul 31 18:43:45.741506 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:45.741460 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqlm2" podUID="0914ed48-d402-4d77-a2e7-ca12693fd0f0" Jul 31 18:43:45.744413 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.744396 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dkbrb" Jul 31 18:43:45.746635 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.746501 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 31 18:43:45.746635 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.746512 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 31 18:43:45.746793 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.746745 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 31 18:43:45.746837 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.746799 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mnl8l\"" Jul 31 18:43:45.747109 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.747090 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.747825 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.747775 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 31 18:43:45.748919 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.748876 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 31 18:43:45.749074 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.749030 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 31 18:43:45.749231 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.749214 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-thmd4\"" Jul 31 18:43:45.750514 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.749887 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 31 18:43:45.750514 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.749973 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 31 18:43:45.750514 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.750090 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 31 18:43:45.750514 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.750135 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 31 18:43:45.750514 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.750259 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fjbvh" Jul 31 18:43:45.752098 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.752058 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8qn59\"" Jul 31 18:43:45.752198 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.752117 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 31 18:43:45.754392 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.753958 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7s5zf" Jul 31 18:43:45.754586 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.754566 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 31 18:43:45.756765 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.756289 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 31 18:43:45.756861 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.756838 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-p2jtr\"" Jul 31 18:43:45.756963 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.756926 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 31 18:43:45.757994 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.757971 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.759029 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.759010 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 31 18:43:45.759829 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.759806 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 31 18:43:45.760301 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.760156 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 31 18:43:45.760460 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.760442 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wqnfg\"" Jul 31 18:43:45.760640 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.760620 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e8223854-5549-4701-b1ce-506ed19d8abe-kubelet-config\") pod \"global-pull-secret-syncer-2rkhw\" (UID: \"e8223854-5549-4701-b1ce-506ed19d8abe\") " pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:43:45.760705 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.760653 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-os-release\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.760705 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.760680 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b47dc\" (UniqueName: \"kubernetes.io/projected/ee8aa68f-72ff-4118-ad38-4da03b305feb-kube-api-access-b47dc\") pod \"network-metrics-daemon-h8n5q\" (UID: \"ee8aa68f-72ff-4118-ad38-4da03b305feb\") " pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:43:45.760811 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.760704 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-host-cni-netd\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.760811 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.760802 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-multus-cni-dir\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.760920 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.760828 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-host-var-lib-kubelet\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.760920 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.760853 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba85de76-d297-4a7b-bb30-9c12572d9811-tmp-dir\") pod \"node-resolver-fjbvh\" (UID: \"ba85de76-d297-4a7b-bb30-9c12572d9811\") " pod="openshift-dns/node-resolver-fjbvh" Jul 31 18:43:45.760920 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.760876 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-systemd-units\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.761075 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.760904 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-host-run-netns\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.761075 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.760949 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-run-systemd\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.761075 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761012 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clg2m\" (UniqueName: \"kubernetes.io/projected/38da7ba5-0ebc-441e-89e3-c265c2da036f-kube-api-access-clg2m\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.761075 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761038 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k697\" (UniqueName: \"kubernetes.io/projected/0914ed48-d402-4d77-a2e7-ca12693fd0f0-kube-api-access-2k697\") pod \"network-check-target-xqlm2\" (UID: \"0914ed48-d402-4d77-a2e7-ca12693fd0f0\") " pod="openshift-network-diagnostics/network-check-target-xqlm2" Jul 31 18:43:45.761075 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761064 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f200fa12-5e9c-4942-886e-cecbc15eaa00-host-slash\") pod \"iptables-alerter-dkbrb\" (UID: \"f200fa12-5e9c-4942-886e-cecbc15eaa00\") " pod="openshift-network-operator/iptables-alerter-dkbrb" Jul 31 18:43:45.761304 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761089 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-host-run-k8s-cni-cncf-io\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.761304 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761115 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-cnibin\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.761304 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761137 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-hostroot\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.761304 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761160 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-etc-openvswitch\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.761304 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761184 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a9871869-ce7a-407a-8bf6-c1f871893ef9-cni-binary-copy\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.761304 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761211 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f200fa12-5e9c-4942-886e-cecbc15eaa00-iptables-alerter-script\") pod \"iptables-alerter-dkbrb\" (UID: \"f200fa12-5e9c-4942-886e-cecbc15eaa00\") " pod="openshift-network-operator/iptables-alerter-dkbrb" Jul 31 18:43:45.761304 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761235 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz8pk\" (UniqueName: \"kubernetes.io/projected/ba85de76-d297-4a7b-bb30-9c12572d9811-kube-api-access-dz8pk\") pod \"node-resolver-fjbvh\" (UID: \"ba85de76-d297-4a7b-bb30-9c12572d9811\") " pod="openshift-dns/node-resolver-fjbvh" Jul 31 18:43:45.761304 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761259 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-node-log\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.761304 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761283 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-host-cni-bin\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.761304 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761307 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/38da7ba5-0ebc-441e-89e3-c265c2da036f-ovnkube-script-lib\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.761807 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761354 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-multus-socket-dir-parent\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.761807 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761384 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-system-cni-dir\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.761807 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761410 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhbbt\" (UniqueName: \"kubernetes.io/projected/a9871869-ce7a-407a-8bf6-c1f871893ef9-kube-api-access-lhbbt\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.761807 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761434 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ba85de76-d297-4a7b-bb30-9c12572d9811-hosts-file\") pod \"node-resolver-fjbvh\" (UID: \"ba85de76-d297-4a7b-bb30-9c12572d9811\") " pod="openshift-dns/node-resolver-fjbvh" Jul 31 18:43:45.761807 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761457 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-host-slash\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.761807 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761482 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-var-lib-openvswitch\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.761807 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761508 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/38da7ba5-0ebc-441e-89e3-c265c2da036f-ovnkube-config\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.761807 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761531 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-multus-conf-dir\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.761807 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761562 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-host-kubelet\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.761807 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761587 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/38da7ba5-0ebc-441e-89e3-c265c2da036f-env-overrides\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.761807 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761611 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/38da7ba5-0ebc-441e-89e3-c265c2da036f-ovn-node-metrics-cert\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.761807 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761636 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a9871869-ce7a-407a-8bf6-c1f871893ef9-multus-daemon-config\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.761807 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761662 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9nbm\" (UniqueName: \"kubernetes.io/projected/f200fa12-5e9c-4942-886e-cecbc15eaa00-kube-api-access-l9nbm\") pod \"iptables-alerter-dkbrb\" (UID: \"f200fa12-5e9c-4942-886e-cecbc15eaa00\") " pod="openshift-network-operator/iptables-alerter-dkbrb" Jul 31 18:43:45.761807 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761685 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-host-run-netns\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.761807 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761711 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8223854-5549-4701-b1ce-506ed19d8abe-original-pull-secret\") pod \"global-pull-secret-syncer-2rkhw\" (UID: \"e8223854-5549-4701-b1ce-506ed19d8abe\") " pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:43:45.761807 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761735 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-host-var-lib-cni-bin\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.761807 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761760 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-host-var-lib-cni-multus\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.762565 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761793 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-host-run-ovn-kubernetes\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.762565 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761827 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e8223854-5549-4701-b1ce-506ed19d8abe-dbus\") pod \"global-pull-secret-syncer-2rkhw\" (UID: \"e8223854-5549-4701-b1ce-506ed19d8abe\") " pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:43:45.762565 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761852 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-host-run-multus-certs\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.762565 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761877 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-etc-kubernetes\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.762565 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761901 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee8aa68f-72ff-4118-ad38-4da03b305feb-metrics-certs\") pod \"network-metrics-daemon-h8n5q\" (UID: \"ee8aa68f-72ff-4118-ad38-4da03b305feb\") " pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:43:45.762565 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761926 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-run-openvswitch\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.762565 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761951 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-run-ovn\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.762565 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.761975 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-log-socket\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.762565 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.762003 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.762565 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.760805 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sbmpv" Jul 31 18:43:45.763052 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.762867 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf2rc" Jul 31 18:43:45.763803 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.763780 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 31 18:43:45.765592 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.765567 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p9mqb" Jul 31 18:43:45.766785 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.766205 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 31 18:43:45.766785 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.766341 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 31 18:43:45.766785 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.766423 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9bt6x\"" Jul 31 18:43:45.766785 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.766428 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 31 18:43:45.766785 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.766561 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 31 18:43:45.766785 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.766594 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 31 18:43:45.766785 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.766647 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wkq7c\"" Jul 31 18:43:45.768025 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.767433 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 31 18:43:45.768025 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.767873 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 31 18:43:45.768432 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.768405 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-27jb9\"" Jul 31 18:43:45.785870 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.785823 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ffj5m" Jul 31 18:43:45.796874 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.796855 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ffj5m" Jul 31 18:43:45.849200 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.849132 2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 31 18:43:45.862917 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.862888 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e8223854-5549-4701-b1ce-506ed19d8abe-dbus\") pod \"global-pull-secret-syncer-2rkhw\" (UID: \"e8223854-5549-4701-b1ce-506ed19d8abe\") " pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:43:45.863027 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.862927 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-run-ovn\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.863027 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.862953 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-log-socket\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.863027 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863013 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cdddb050-55be-48e1-9a8e-df26711e4416-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p9mqb\" (UID: \"cdddb050-55be-48e1-9a8e-df26711e4416\") " pod="openshift-multus/multus-additional-cni-plugins-p9mqb" Jul 31 18:43:45.863179 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863042 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-etc-sysctl-conf\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.863179 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863044 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-log-socket\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.863179 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863070 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e8223854-5549-4701-b1ce-506ed19d8abe-kubelet-config\") pod \"global-pull-secret-syncer-2rkhw\" (UID: \"e8223854-5549-4701-b1ce-506ed19d8abe\") " pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:43:45.863179 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863103 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-run-ovn\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.863179 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863128 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-os-release\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.863179 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863136 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e8223854-5549-4701-b1ce-506ed19d8abe-kubelet-config\") pod \"global-pull-secret-syncer-2rkhw\" (UID: \"e8223854-5549-4701-b1ce-506ed19d8abe\") " pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:43:45.863179 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863165 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b47dc\" (UniqueName: \"kubernetes.io/projected/ee8aa68f-72ff-4118-ad38-4da03b305feb-kube-api-access-b47dc\") pod \"network-metrics-daemon-h8n5q\" (UID: \"ee8aa68f-72ff-4118-ad38-4da03b305feb\") " pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:43:45.863586 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863196 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f84e1304-da65-4c5c-ad8d-44e1c8b0808b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pf2rc\" (UID: \"f84e1304-da65-4c5c-ad8d-44e1c8b0808b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf2rc" Jul 31 18:43:45.863586 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863198 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-os-release\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.863586 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863227 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f84e1304-da65-4c5c-ad8d-44e1c8b0808b-registration-dir\") pod \"aws-ebs-csi-driver-node-pf2rc\" (UID: \"f84e1304-da65-4c5c-ad8d-44e1c8b0808b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf2rc" Jul 31 18:43:45.863586 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863276 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-host\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.863586 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863321 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-multus-cni-dir\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.863586 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863326 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e8223854-5549-4701-b1ce-506ed19d8abe-dbus\") pod \"global-pull-secret-syncer-2rkhw\" (UID: \"e8223854-5549-4701-b1ce-506ed19d8abe\") " pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:43:45.863586 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863386 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-multus-cni-dir\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.863586 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863413 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba85de76-d297-4a7b-bb30-9c12572d9811-tmp-dir\") pod \"node-resolver-fjbvh\" (UID: \"ba85de76-d297-4a7b-bb30-9c12572d9811\") " pod="openshift-dns/node-resolver-fjbvh" Jul 31 18:43:45.863586 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863441 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f84e1304-da65-4c5c-ad8d-44e1c8b0808b-socket-dir\") pod \"aws-ebs-csi-driver-node-pf2rc\" (UID: \"f84e1304-da65-4c5c-ad8d-44e1c8b0808b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf2rc" Jul 31 18:43:45.863586 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863464 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ffecd04e-7d64-429f-95a8-20647557526d-agent-certs\") pod \"konnectivity-agent-7s5zf\" (UID: \"ffecd04e-7d64-429f-95a8-20647557526d\") " pod="kube-system/konnectivity-agent-7s5zf" Jul 31 18:43:45.863586 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863496 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-etc-kubernetes\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.863586 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863521 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq5nh\" (UniqueName: \"kubernetes.io/projected/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-kube-api-access-dq5nh\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.863586 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863544 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k697\" (UniqueName: \"kubernetes.io/projected/0914ed48-d402-4d77-a2e7-ca12693fd0f0-kube-api-access-2k697\") pod \"network-check-target-xqlm2\" (UID: \"0914ed48-d402-4d77-a2e7-ca12693fd0f0\") " pod="openshift-network-diagnostics/network-check-target-xqlm2" Jul 31 18:43:45.863586 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863568 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-hostroot\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.863586 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863589 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f84e1304-da65-4c5c-ad8d-44e1c8b0808b-device-dir\") pod \"aws-ebs-csi-driver-node-pf2rc\" (UID: \"f84e1304-da65-4c5c-ad8d-44e1c8b0808b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf2rc" Jul 31 18:43:45.864365 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863609 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f84e1304-da65-4c5c-ad8d-44e1c8b0808b-sys-fs\") pod \"aws-ebs-csi-driver-node-pf2rc\" (UID: \"f84e1304-da65-4c5c-ad8d-44e1c8b0808b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf2rc" Jul 31 18:43:45.864365 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863630 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-node-log\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.864365 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863646 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-hostroot\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.864365 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863651 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/38da7ba5-0ebc-441e-89e3-c265c2da036f-ovnkube-script-lib\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.864365 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863690 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ba85de76-d297-4a7b-bb30-9c12572d9811-hosts-file\") pod \"node-resolver-fjbvh\" (UID: \"ba85de76-d297-4a7b-bb30-9c12572d9811\") " pod="openshift-dns/node-resolver-fjbvh" Jul 31 18:43:45.864365 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863723 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-host-slash\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.864365 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863756 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-var-lib-openvswitch\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.864365 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863760 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba85de76-d297-4a7b-bb30-9c12572d9811-tmp-dir\") pod \"node-resolver-fjbvh\" (UID: \"ba85de76-d297-4a7b-bb30-9c12572d9811\") " pod="openshift-dns/node-resolver-fjbvh" Jul 31 18:43:45.864365 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863784 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/38da7ba5-0ebc-441e-89e3-c265c2da036f-ovnkube-config\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.864365 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863761 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-node-log\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.864365 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863804 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ba85de76-d297-4a7b-bb30-9c12572d9811-hosts-file\") pod \"node-resolver-fjbvh\" (UID: \"ba85de76-d297-4a7b-bb30-9c12572d9811\") " pod="openshift-dns/node-resolver-fjbvh" Jul 31 18:43:45.864365 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863838 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-host-slash\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.864365 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863838 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-var-lib-openvswitch\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.864365 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863862 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/38da7ba5-0ebc-441e-89e3-c265c2da036f-env-overrides\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.864365 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863908 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzrjn\" (UniqueName: \"kubernetes.io/projected/fa723eb5-7c9f-4676-aeda-50b07733ff96-kube-api-access-jzrjn\") pod \"node-ca-sbmpv\" (UID: \"fa723eb5-7c9f-4676-aeda-50b07733ff96\") " pod="openshift-image-registry/node-ca-sbmpv" Jul 31 18:43:45.864365 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863937 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-host-kubelet\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.864365 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863979 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-host-kubelet\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.864365 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.863979 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ffecd04e-7d64-429f-95a8-20647557526d-konnectivity-ca\") pod \"konnectivity-agent-7s5zf\" (UID: \"ffecd04e-7d64-429f-95a8-20647557526d\") " pod="kube-system/konnectivity-agent-7s5zf" Jul 31 18:43:45.865162 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864016 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-etc-sysconfig\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.865162 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864044 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a9871869-ce7a-407a-8bf6-c1f871893ef9-multus-daemon-config\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.865162 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864070 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-host-var-lib-cni-multus\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.865162 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864096 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-host-run-ovn-kubernetes\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.865162 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864125 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.865162 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864153 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsbbb\" (UniqueName: \"kubernetes.io/projected/f84e1304-da65-4c5c-ad8d-44e1c8b0808b-kube-api-access-xsbbb\") pod \"aws-ebs-csi-driver-node-pf2rc\" (UID: \"f84e1304-da65-4c5c-ad8d-44e1c8b0808b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf2rc" Jul 31 18:43:45.865162 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864180 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdddb050-55be-48e1-9a8e-df26711e4416-system-cni-dir\") pod \"multus-additional-cni-plugins-p9mqb\" (UID: \"cdddb050-55be-48e1-9a8e-df26711e4416\") " pod="openshift-multus/multus-additional-cni-plugins-p9mqb" Jul 31 18:43:45.865162 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864206 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-host-run-multus-certs\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.865162 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864231 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-etc-kubernetes\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.865162 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864236 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-host-run-ovn-kubernetes\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.865162 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864272 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/38da7ba5-0ebc-441e-89e3-c265c2da036f-ovnkube-script-lib\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.865162 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864301 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.865162 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864334 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/38da7ba5-0ebc-441e-89e3-c265c2da036f-ovnkube-config\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.865162 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:45.864349 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 18:43:45.865162 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864256 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee8aa68f-72ff-4118-ad38-4da03b305feb-metrics-certs\") pod \"network-metrics-daemon-h8n5q\" (UID: \"ee8aa68f-72ff-4118-ad38-4da03b305feb\") " pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:43:45.865162 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864367 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-host-run-multus-certs\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.865162 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864379 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-host-var-lib-cni-multus\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.865968 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864390 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-run-openvswitch\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.865968 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864408 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-etc-kubernetes\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.865968 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864407 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/38da7ba5-0ebc-441e-89e3-c265c2da036f-env-overrides\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.865968 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:45.864426 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee8aa68f-72ff-4118-ad38-4da03b305feb-metrics-certs podName:ee8aa68f-72ff-4118-ad38-4da03b305feb nodeName:}" failed. No retries permitted until 2026-07-31 18:43:46.364401918 +0000 UTC m=+3.002391718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee8aa68f-72ff-4118-ad38-4da03b305feb-metrics-certs") pod "network-metrics-daemon-h8n5q" (UID: "ee8aa68f-72ff-4118-ad38-4da03b305feb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 18:43:45.865968 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864438 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-run-openvswitch\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.865968 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864450 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cdddb050-55be-48e1-9a8e-df26711e4416-os-release\") pod \"multus-additional-cni-plugins-p9mqb\" (UID: \"cdddb050-55be-48e1-9a8e-df26711e4416\") " pod="openshift-multus/multus-additional-cni-plugins-p9mqb" Jul 31 18:43:45.865968 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864483 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-host-cni-netd\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.865968 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864510 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cdddb050-55be-48e1-9a8e-df26711e4416-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p9mqb\" (UID: \"cdddb050-55be-48e1-9a8e-df26711e4416\") " pod="openshift-multus/multus-additional-cni-plugins-p9mqb" Jul 31 18:43:45.865968 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864536 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-host-cni-netd\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.865968 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864538 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-etc-systemd\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.865968 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864580 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-run\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.865968 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864603 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-etc-tuned\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.865968 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864630 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-host-var-lib-kubelet\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.865968 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864657 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-systemd-units\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.865968 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864667 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-host-var-lib-kubelet\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.865968 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864684 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-host-run-netns\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.866792 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864717 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-systemd-units\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.866792 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864736 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-run-systemd\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.866792 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864690 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a9871869-ce7a-407a-8bf6-c1f871893ef9-multus-daemon-config\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.866792 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864763 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f84e1304-da65-4c5c-ad8d-44e1c8b0808b-etc-selinux\") pod \"aws-ebs-csi-driver-node-pf2rc\" (UID: \"f84e1304-da65-4c5c-ad8d-44e1c8b0808b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf2rc" Jul 31 18:43:45.866792 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864817 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-run-systemd\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.866792 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864854 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-etc-modprobe-d\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.866792 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864875 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-host-run-netns\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.866792 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864882 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-lib-modules\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.866792 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864930 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f200fa12-5e9c-4942-886e-cecbc15eaa00-host-slash\") pod \"iptables-alerter-dkbrb\" (UID: \"f200fa12-5e9c-4942-886e-cecbc15eaa00\") " pod="openshift-network-operator/iptables-alerter-dkbrb" Jul 31 18:43:45.866792 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864955 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-host-run-k8s-cni-cncf-io\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.866792 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.864994 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-cnibin\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.866792 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.865007 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-host-run-k8s-cni-cncf-io\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.866792 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.865032 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-etc-openvswitch\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.866792 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.865049 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f200fa12-5e9c-4942-886e-cecbc15eaa00-host-slash\") pod \"iptables-alerter-dkbrb\" (UID: \"f200fa12-5e9c-4942-886e-cecbc15eaa00\") " pod="openshift-network-operator/iptables-alerter-dkbrb" Jul 31 18:43:45.866792 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.865060 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-var-lib-kubelet\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.866792 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.865085 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-tmp\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.866792 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.865099 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-cnibin\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.867634 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.865110 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fa723eb5-7c9f-4676-aeda-50b07733ff96-serviceca\") pod \"node-ca-sbmpv\" (UID: \"fa723eb5-7c9f-4676-aeda-50b07733ff96\") " pod="openshift-image-registry/node-ca-sbmpv" Jul 31 18:43:45.867634 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.865135 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a9871869-ce7a-407a-8bf6-c1f871893ef9-cni-binary-copy\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.867634 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.865144 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-etc-openvswitch\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.867634 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.865160 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f200fa12-5e9c-4942-886e-cecbc15eaa00-iptables-alerter-script\") pod \"iptables-alerter-dkbrb\" (UID: \"f200fa12-5e9c-4942-886e-cecbc15eaa00\") " pod="openshift-network-operator/iptables-alerter-dkbrb" Jul 31 18:43:45.867634 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.865622 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz8pk\" (UniqueName: \"kubernetes.io/projected/ba85de76-d297-4a7b-bb30-9c12572d9811-kube-api-access-dz8pk\") pod \"node-resolver-fjbvh\" (UID: \"ba85de76-d297-4a7b-bb30-9c12572d9811\") " pod="openshift-dns/node-resolver-fjbvh" Jul 31 18:43:45.867634 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.865647 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-host-cni-bin\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.867634 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.865654 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a9871869-ce7a-407a-8bf6-c1f871893ef9-cni-binary-copy\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.867634 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.865672 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cdddb050-55be-48e1-9a8e-df26711e4416-cnibin\") pod \"multus-additional-cni-plugins-p9mqb\" (UID: \"cdddb050-55be-48e1-9a8e-df26711e4416\") " pod="openshift-multus/multus-additional-cni-plugins-p9mqb" Jul 31 18:43:45.867634 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.865700 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cdddb050-55be-48e1-9a8e-df26711e4416-cni-binary-copy\") pod \"multus-additional-cni-plugins-p9mqb\" (UID: \"cdddb050-55be-48e1-9a8e-df26711e4416\") " pod="openshift-multus/multus-additional-cni-plugins-p9mqb" Jul 31 18:43:45.867634 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.865724 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-sys\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.867634 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.865748 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-multus-socket-dir-parent\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.867634 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.865786 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-system-cni-dir\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.867634 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.865811 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhbbt\" (UniqueName: \"kubernetes.io/projected/a9871869-ce7a-407a-8bf6-c1f871893ef9-kube-api-access-lhbbt\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.867634 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.865836 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/38da7ba5-0ebc-441e-89e3-c265c2da036f-ovn-node-metrics-cert\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.867634 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.865861 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4xzd\" (UniqueName: \"kubernetes.io/projected/cdddb050-55be-48e1-9a8e-df26711e4416-kube-api-access-f4xzd\") pod \"multus-additional-cni-plugins-p9mqb\" (UID: \"cdddb050-55be-48e1-9a8e-df26711e4416\") " pod="openshift-multus/multus-additional-cni-plugins-p9mqb" Jul 31 18:43:45.867634 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.865900 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa723eb5-7c9f-4676-aeda-50b07733ff96-host\") pod \"node-ca-sbmpv\" (UID: \"fa723eb5-7c9f-4676-aeda-50b07733ff96\") " pod="openshift-image-registry/node-ca-sbmpv" Jul 31 18:43:45.867634 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.865933 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-multus-socket-dir-parent\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.868573 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.865945 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f200fa12-5e9c-4942-886e-cecbc15eaa00-iptables-alerter-script\") pod \"iptables-alerter-dkbrb\" (UID: \"f200fa12-5e9c-4942-886e-cecbc15eaa00\") " pod="openshift-network-operator/iptables-alerter-dkbrb" Jul 31 18:43:45.868573 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.865974 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/38da7ba5-0ebc-441e-89e3-c265c2da036f-host-cni-bin\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.868573 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.865933 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-multus-conf-dir\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.868573 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.865973 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-multus-conf-dir\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.868573 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.866024 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-etc-sysctl-d\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.868573 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.866054 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9nbm\" (UniqueName: \"kubernetes.io/projected/f200fa12-5e9c-4942-886e-cecbc15eaa00-kube-api-access-l9nbm\") pod \"iptables-alerter-dkbrb\" (UID: \"f200fa12-5e9c-4942-886e-cecbc15eaa00\") " pod="openshift-network-operator/iptables-alerter-dkbrb" Jul 31 18:43:45.868573 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.866081 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-host-run-netns\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.868573 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.866106 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8223854-5549-4701-b1ce-506ed19d8abe-original-pull-secret\") pod \"global-pull-secret-syncer-2rkhw\" (UID: \"e8223854-5549-4701-b1ce-506ed19d8abe\") " pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:43:45.868573 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.866131 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-host-var-lib-cni-bin\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.868573 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.866155 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clg2m\" (UniqueName: \"kubernetes.io/projected/38da7ba5-0ebc-441e-89e3-c265c2da036f-kube-api-access-clg2m\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.868573 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.866182 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cdddb050-55be-48e1-9a8e-df26711e4416-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p9mqb\" (UID: \"cdddb050-55be-48e1-9a8e-df26711e4416\") " pod="openshift-multus/multus-additional-cni-plugins-p9mqb" Jul 31 18:43:45.868573 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.866232 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-host-run-netns\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.868573 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.866286 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-host-var-lib-cni-bin\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.868573 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.866301 2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 31 18:43:45.868573 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.866358 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9871869-ce7a-407a-8bf6-c1f871893ef9-system-cni-dir\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.868573 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:45.866435 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 18:43:45.868573 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:45.866485 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8223854-5549-4701-b1ce-506ed19d8abe-original-pull-secret podName:e8223854-5549-4701-b1ce-506ed19d8abe nodeName:}" failed. No retries permitted until 2026-07-31 18:43:46.366465695 +0000 UTC m=+3.004455490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e8223854-5549-4701-b1ce-506ed19d8abe-original-pull-secret") pod "global-pull-secret-syncer-2rkhw" (UID: "e8223854-5549-4701-b1ce-506ed19d8abe") : object "kube-system"/"original-pull-secret" not registered Jul 31 18:43:45.872609 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.872588 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/38da7ba5-0ebc-441e-89e3-c265c2da036f-ovn-node-metrics-cert\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.872980 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:45.872965 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 18:43:45.873062 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:45.872987 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 18:43:45.873062 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:45.873000 2565 projected.go:196] Error preparing data for projected volume kube-api-access-2k697 for pod openshift-network-diagnostics/network-check-target-xqlm2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 18:43:45.873062 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:45.873042 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0914ed48-d402-4d77-a2e7-ca12693fd0f0-kube-api-access-2k697 podName:0914ed48-d402-4d77-a2e7-ca12693fd0f0 nodeName:}" failed. No retries permitted until 2026-07-31 18:43:46.373030205 +0000 UTC m=+3.011019985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2k697" (UniqueName: "kubernetes.io/projected/0914ed48-d402-4d77-a2e7-ca12693fd0f0-kube-api-access-2k697") pod "network-check-target-xqlm2" (UID: "0914ed48-d402-4d77-a2e7-ca12693fd0f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 18:43:45.877573 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.877548 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhbbt\" (UniqueName: \"kubernetes.io/projected/a9871869-ce7a-407a-8bf6-c1f871893ef9-kube-api-access-lhbbt\") pod \"multus-vbbdc\" (UID: \"a9871869-ce7a-407a-8bf6-c1f871893ef9\") " pod="openshift-multus/multus-vbbdc" Jul 31 18:43:45.878839 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.878808 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b47dc\" (UniqueName: \"kubernetes.io/projected/ee8aa68f-72ff-4118-ad38-4da03b305feb-kube-api-access-b47dc\") pod \"network-metrics-daemon-h8n5q\" (UID: \"ee8aa68f-72ff-4118-ad38-4da03b305feb\") " pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:43:45.880117 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.880095 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clg2m\" (UniqueName: \"kubernetes.io/projected/38da7ba5-0ebc-441e-89e3-c265c2da036f-kube-api-access-clg2m\") pod \"ovnkube-node-fxm9n\" (UID: \"38da7ba5-0ebc-441e-89e3-c265c2da036f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:45.880200 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.880163 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz8pk\" (UniqueName: \"kubernetes.io/projected/ba85de76-d297-4a7b-bb30-9c12572d9811-kube-api-access-dz8pk\") pod \"node-resolver-fjbvh\" (UID: \"ba85de76-d297-4a7b-bb30-9c12572d9811\") " pod="openshift-dns/node-resolver-fjbvh" Jul 31 18:43:45.880200 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.880186 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9nbm\" (UniqueName: \"kubernetes.io/projected/f200fa12-5e9c-4942-886e-cecbc15eaa00-kube-api-access-l9nbm\") pod \"iptables-alerter-dkbrb\" (UID: \"f200fa12-5e9c-4942-886e-cecbc15eaa00\") " pod="openshift-network-operator/iptables-alerter-dkbrb" Jul 31 18:43:45.884678 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.884633 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal" event={"ID":"6277a79a23fe911d619a284a778ea843","Type":"ContainerStarted","Data":"573452a1ca41c4d0d4f287397555223254e0a8b0a4174e6b7ed58da4fc261bc4"} Jul 31 18:43:45.885718 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.885694 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-39.ec2.internal" event={"ID":"6781281007e7ddfaa5111df980719fef","Type":"ContainerStarted","Data":"71c83eef61737a952e0d1568897677ca143013ad1de8891edf55ddc92c9b6d59"} Jul 31 18:43:45.966513 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.966477 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-var-lib-kubelet\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.966694 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.966522 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-tmp\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.966694 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.966546 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fa723eb5-7c9f-4676-aeda-50b07733ff96-serviceca\") pod \"node-ca-sbmpv\" (UID: \"fa723eb5-7c9f-4676-aeda-50b07733ff96\") " pod="openshift-image-registry/node-ca-sbmpv" Jul 31 18:43:45.966694 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.966574 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cdddb050-55be-48e1-9a8e-df26711e4416-cnibin\") pod \"multus-additional-cni-plugins-p9mqb\" (UID: \"cdddb050-55be-48e1-9a8e-df26711e4416\") " pod="openshift-multus/multus-additional-cni-plugins-p9mqb" Jul 31 18:43:45.966694 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.966597 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cdddb050-55be-48e1-9a8e-df26711e4416-cni-binary-copy\") pod \"multus-additional-cni-plugins-p9mqb\" (UID: \"cdddb050-55be-48e1-9a8e-df26711e4416\") " pod="openshift-multus/multus-additional-cni-plugins-p9mqb" Jul 31 18:43:45.966694 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.966619 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-sys\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.966694 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.966645 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4xzd\" (UniqueName: \"kubernetes.io/projected/cdddb050-55be-48e1-9a8e-df26711e4416-kube-api-access-f4xzd\") pod \"multus-additional-cni-plugins-p9mqb\" (UID: \"cdddb050-55be-48e1-9a8e-df26711e4416\") " pod="openshift-multus/multus-additional-cni-plugins-p9mqb" Jul 31 18:43:45.966694 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.966669 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa723eb5-7c9f-4676-aeda-50b07733ff96-host\") pod \"node-ca-sbmpv\" (UID: \"fa723eb5-7c9f-4676-aeda-50b07733ff96\") " pod="openshift-image-registry/node-ca-sbmpv" Jul 31 18:43:45.966694 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.966693 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-etc-sysctl-d\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.967116 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.966697 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-var-lib-kubelet\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.967116 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.966739 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cdddb050-55be-48e1-9a8e-df26711e4416-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p9mqb\" (UID: \"cdddb050-55be-48e1-9a8e-df26711e4416\") " pod="openshift-multus/multus-additional-cni-plugins-p9mqb" Jul 31 18:43:45.967116 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.966768 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cdddb050-55be-48e1-9a8e-df26711e4416-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p9mqb\" (UID: \"cdddb050-55be-48e1-9a8e-df26711e4416\") " pod="openshift-multus/multus-additional-cni-plugins-p9mqb" Jul 31 18:43:45.967116 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.966796 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-etc-sysctl-conf\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.967116 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.966822 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-sys\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.967116 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.966827 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f84e1304-da65-4c5c-ad8d-44e1c8b0808b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pf2rc\" (UID: \"f84e1304-da65-4c5c-ad8d-44e1c8b0808b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf2rc" Jul 31 18:43:45.967116 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.966891 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f84e1304-da65-4c5c-ad8d-44e1c8b0808b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pf2rc\" (UID: \"f84e1304-da65-4c5c-ad8d-44e1c8b0808b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf2rc" Jul 31 18:43:45.967116 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.966925 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f84e1304-da65-4c5c-ad8d-44e1c8b0808b-registration-dir\") pod \"aws-ebs-csi-driver-node-pf2rc\" (UID: \"f84e1304-da65-4c5c-ad8d-44e1c8b0808b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf2rc" Jul 31 18:43:45.967116 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.966952 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-host\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.967116 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.966768 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cdddb050-55be-48e1-9a8e-df26711e4416-cnibin\") pod \"multus-additional-cni-plugins-p9mqb\" (UID: \"cdddb050-55be-48e1-9a8e-df26711e4416\") " pod="openshift-multus/multus-additional-cni-plugins-p9mqb" Jul 31 18:43:45.967116 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.966979 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f84e1304-da65-4c5c-ad8d-44e1c8b0808b-socket-dir\") pod \"aws-ebs-csi-driver-node-pf2rc\" (UID: \"f84e1304-da65-4c5c-ad8d-44e1c8b0808b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf2rc" Jul 31 18:43:45.967116 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967028 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ffecd04e-7d64-429f-95a8-20647557526d-agent-certs\") pod \"konnectivity-agent-7s5zf\" (UID: \"ffecd04e-7d64-429f-95a8-20647557526d\") " pod="kube-system/konnectivity-agent-7s5zf" Jul 31 18:43:45.967116 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967049 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-etc-kubernetes\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.967116 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967072 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq5nh\" (UniqueName: \"kubernetes.io/projected/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-kube-api-access-dq5nh\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.967116 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967087 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f84e1304-da65-4c5c-ad8d-44e1c8b0808b-socket-dir\") pod \"aws-ebs-csi-driver-node-pf2rc\" (UID: \"f84e1304-da65-4c5c-ad8d-44e1c8b0808b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf2rc" Jul 31 18:43:45.967831 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967130 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa723eb5-7c9f-4676-aeda-50b07733ff96-host\") pod \"node-ca-sbmpv\" (UID: \"fa723eb5-7c9f-4676-aeda-50b07733ff96\") " pod="openshift-image-registry/node-ca-sbmpv" Jul 31 18:43:45.967831 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967154 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cdddb050-55be-48e1-9a8e-df26711e4416-cni-binary-copy\") pod \"multus-additional-cni-plugins-p9mqb\" (UID: \"cdddb050-55be-48e1-9a8e-df26711e4416\") " pod="openshift-multus/multus-additional-cni-plugins-p9mqb" Jul 31 18:43:45.967831 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967204 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-etc-sysctl-d\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.967831 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967267 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-etc-sysctl-conf\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.967831 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967266 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cdddb050-55be-48e1-9a8e-df26711e4416-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p9mqb\" (UID: \"cdddb050-55be-48e1-9a8e-df26711e4416\") " pod="openshift-multus/multus-additional-cni-plugins-p9mqb" Jul 31 18:43:45.967831 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967345 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f84e1304-da65-4c5c-ad8d-44e1c8b0808b-registration-dir\") pod \"aws-ebs-csi-driver-node-pf2rc\" (UID: \"f84e1304-da65-4c5c-ad8d-44e1c8b0808b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf2rc" Jul 31 18:43:45.967831 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967398 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-host\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.967831 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967448 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-etc-kubernetes\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.967831 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967495 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f84e1304-da65-4c5c-ad8d-44e1c8b0808b-device-dir\") pod \"aws-ebs-csi-driver-node-pf2rc\" (UID: \"f84e1304-da65-4c5c-ad8d-44e1c8b0808b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf2rc" Jul 31 18:43:45.967831 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967528 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f84e1304-da65-4c5c-ad8d-44e1c8b0808b-sys-fs\") pod \"aws-ebs-csi-driver-node-pf2rc\" (UID: \"f84e1304-da65-4c5c-ad8d-44e1c8b0808b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf2rc" Jul 31 18:43:45.967831 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967530 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fa723eb5-7c9f-4676-aeda-50b07733ff96-serviceca\") pod \"node-ca-sbmpv\" (UID: \"fa723eb5-7c9f-4676-aeda-50b07733ff96\") " pod="openshift-image-registry/node-ca-sbmpv" Jul 31 18:43:45.967831 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967580 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzrjn\" (UniqueName: \"kubernetes.io/projected/fa723eb5-7c9f-4676-aeda-50b07733ff96-kube-api-access-jzrjn\") pod \"node-ca-sbmpv\" (UID: \"fa723eb5-7c9f-4676-aeda-50b07733ff96\") " pod="openshift-image-registry/node-ca-sbmpv" Jul 31 18:43:45.967831 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967595 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f84e1304-da65-4c5c-ad8d-44e1c8b0808b-sys-fs\") pod \"aws-ebs-csi-driver-node-pf2rc\" (UID: \"f84e1304-da65-4c5c-ad8d-44e1c8b0808b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf2rc" Jul 31 18:43:45.967831 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967602 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cdddb050-55be-48e1-9a8e-df26711e4416-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p9mqb\" (UID: \"cdddb050-55be-48e1-9a8e-df26711e4416\") " pod="openshift-multus/multus-additional-cni-plugins-p9mqb" Jul 31 18:43:45.967831 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967612 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ffecd04e-7d64-429f-95a8-20647557526d-konnectivity-ca\") pod \"konnectivity-agent-7s5zf\" (UID: \"ffecd04e-7d64-429f-95a8-20647557526d\") " pod="kube-system/konnectivity-agent-7s5zf" Jul 31 18:43:45.967831 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967639 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-etc-sysconfig\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.967831 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967647 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f84e1304-da65-4c5c-ad8d-44e1c8b0808b-device-dir\") pod \"aws-ebs-csi-driver-node-pf2rc\" (UID: \"f84e1304-da65-4c5c-ad8d-44e1c8b0808b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf2rc" Jul 31 18:43:45.968768 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967676 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsbbb\" (UniqueName: \"kubernetes.io/projected/f84e1304-da65-4c5c-ad8d-44e1c8b0808b-kube-api-access-xsbbb\") pod \"aws-ebs-csi-driver-node-pf2rc\" (UID: \"f84e1304-da65-4c5c-ad8d-44e1c8b0808b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf2rc" Jul 31 18:43:45.968768 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967695 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-etc-sysconfig\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.968768 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967702 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdddb050-55be-48e1-9a8e-df26711e4416-system-cni-dir\") pod \"multus-additional-cni-plugins-p9mqb\" (UID: \"cdddb050-55be-48e1-9a8e-df26711e4416\") " pod="openshift-multus/multus-additional-cni-plugins-p9mqb" Jul 31 18:43:45.968768 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967742 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cdddb050-55be-48e1-9a8e-df26711e4416-os-release\") pod \"multus-additional-cni-plugins-p9mqb\" (UID: \"cdddb050-55be-48e1-9a8e-df26711e4416\") " pod="openshift-multus/multus-additional-cni-plugins-p9mqb" Jul 31 18:43:45.968768 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967770 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cdddb050-55be-48e1-9a8e-df26711e4416-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p9mqb\" (UID: \"cdddb050-55be-48e1-9a8e-df26711e4416\") " pod="openshift-multus/multus-additional-cni-plugins-p9mqb" Jul 31 18:43:45.968768 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967797 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-etc-systemd\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.968768 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967821 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-run\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.968768 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967870 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-run\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.968768 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967889 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cdddb050-55be-48e1-9a8e-df26711e4416-os-release\") pod \"multus-additional-cni-plugins-p9mqb\" (UID: \"cdddb050-55be-48e1-9a8e-df26711e4416\") " pod="openshift-multus/multus-additional-cni-plugins-p9mqb" Jul 31 18:43:45.968768 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967927 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-etc-tuned\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.968768 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967960 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdddb050-55be-48e1-9a8e-df26711e4416-system-cni-dir\") pod \"multus-additional-cni-plugins-p9mqb\" (UID: \"cdddb050-55be-48e1-9a8e-df26711e4416\") " pod="openshift-multus/multus-additional-cni-plugins-p9mqb" Jul 31 18:43:45.968768 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.967982 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f84e1304-da65-4c5c-ad8d-44e1c8b0808b-etc-selinux\") pod \"aws-ebs-csi-driver-node-pf2rc\" (UID: \"f84e1304-da65-4c5c-ad8d-44e1c8b0808b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf2rc" Jul 31 18:43:45.968768 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.968007 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-etc-modprobe-d\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.968768 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.968065 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-lib-modules\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.968768 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.968196 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-lib-modules\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.968768 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.968195 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ffecd04e-7d64-429f-95a8-20647557526d-konnectivity-ca\") pod \"konnectivity-agent-7s5zf\" (UID: \"ffecd04e-7d64-429f-95a8-20647557526d\") " pod="kube-system/konnectivity-agent-7s5zf" Jul 31 18:43:45.968768 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.968253 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-etc-systemd\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.969658 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.968262 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f84e1304-da65-4c5c-ad8d-44e1c8b0808b-etc-selinux\") pod \"aws-ebs-csi-driver-node-pf2rc\" (UID: \"f84e1304-da65-4c5c-ad8d-44e1c8b0808b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf2rc" Jul 31 18:43:45.969658 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.968349 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cdddb050-55be-48e1-9a8e-df26711e4416-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p9mqb\" (UID: \"cdddb050-55be-48e1-9a8e-df26711e4416\") " pod="openshift-multus/multus-additional-cni-plugins-p9mqb" Jul 31 18:43:45.969658 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.968358 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-etc-modprobe-d\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.969823 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.969727 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-tmp\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.970407 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.970123 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ffecd04e-7d64-429f-95a8-20647557526d-agent-certs\") pod \"konnectivity-agent-7s5zf\" (UID: \"ffecd04e-7d64-429f-95a8-20647557526d\") " pod="kube-system/konnectivity-agent-7s5zf" Jul 31 18:43:45.970407 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.970277 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-etc-tuned\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.976649 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.976618 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq5nh\" (UniqueName: \"kubernetes.io/projected/e1eb24da-aa9d-44e8-b12f-6e674dfd589f-kube-api-access-dq5nh\") pod \"tuned-jh7cb\" (UID: \"e1eb24da-aa9d-44e8-b12f-6e674dfd589f\") " pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:45.977089 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.977063 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsbbb\" (UniqueName: \"kubernetes.io/projected/f84e1304-da65-4c5c-ad8d-44e1c8b0808b-kube-api-access-xsbbb\") pod \"aws-ebs-csi-driver-node-pf2rc\" (UID: \"f84e1304-da65-4c5c-ad8d-44e1c8b0808b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf2rc" Jul 31 18:43:45.977204 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.977152 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzrjn\" (UniqueName: \"kubernetes.io/projected/fa723eb5-7c9f-4676-aeda-50b07733ff96-kube-api-access-jzrjn\") pod \"node-ca-sbmpv\" (UID: \"fa723eb5-7c9f-4676-aeda-50b07733ff96\") " pod="openshift-image-registry/node-ca-sbmpv" Jul 31 18:43:45.977204 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:45.977186 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4xzd\" (UniqueName: \"kubernetes.io/projected/cdddb050-55be-48e1-9a8e-df26711e4416-kube-api-access-f4xzd\") pod \"multus-additional-cni-plugins-p9mqb\" (UID: \"cdddb050-55be-48e1-9a8e-df26711e4416\") " pod="openshift-multus/multus-additional-cni-plugins-p9mqb" Jul 31 18:43:46.050325 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:46.050254 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 18:43:46.050325 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:46.050284 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vbbdc" Jul 31 18:43:46.067566 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:46.067535 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dkbrb" Jul 31 18:43:46.076377 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:46.076358 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:43:46.083342 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:46.083320 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fjbvh" Jul 31 18:43:46.091052 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:46.091036 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7s5zf" Jul 31 18:43:46.099890 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:46.099873 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" Jul 31 18:43:46.107562 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:46.107540 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sbmpv" Jul 31 18:43:46.116495 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:46.116473 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf2rc" Jul 31 18:43:46.123308 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:46.123288 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p9mqb" Jul 31 18:43:46.370847 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:46.370765 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8223854-5549-4701-b1ce-506ed19d8abe-original-pull-secret\") pod \"global-pull-secret-syncer-2rkhw\" (UID: \"e8223854-5549-4701-b1ce-506ed19d8abe\") " pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:43:46.371003 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:46.370853 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee8aa68f-72ff-4118-ad38-4da03b305feb-metrics-certs\") pod \"network-metrics-daemon-h8n5q\" (UID: \"ee8aa68f-72ff-4118-ad38-4da03b305feb\") " pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:43:46.371003 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:46.370962 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 18:43:46.371003 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:46.370993 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 18:43:46.371146 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:46.371049 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8223854-5549-4701-b1ce-506ed19d8abe-original-pull-secret podName:e8223854-5549-4701-b1ce-506ed19d8abe nodeName:}" failed. No retries permitted until 2026-07-31 18:43:47.371024419 +0000 UTC m=+4.009014202 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e8223854-5549-4701-b1ce-506ed19d8abe-original-pull-secret") pod "global-pull-secret-syncer-2rkhw" (UID: "e8223854-5549-4701-b1ce-506ed19d8abe") : object "kube-system"/"original-pull-secret" not registered Jul 31 18:43:46.371146 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:46.371071 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee8aa68f-72ff-4118-ad38-4da03b305feb-metrics-certs podName:ee8aa68f-72ff-4118-ad38-4da03b305feb nodeName:}" failed. No retries permitted until 2026-07-31 18:43:47.371059603 +0000 UTC m=+4.009049390 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee8aa68f-72ff-4118-ad38-4da03b305feb-metrics-certs") pod "network-metrics-daemon-h8n5q" (UID: "ee8aa68f-72ff-4118-ad38-4da03b305feb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 18:43:46.471619 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:46.471572 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k697\" (UniqueName: \"kubernetes.io/projected/0914ed48-d402-4d77-a2e7-ca12693fd0f0-kube-api-access-2k697\") pod \"network-check-target-xqlm2\" (UID: \"0914ed48-d402-4d77-a2e7-ca12693fd0f0\") " pod="openshift-network-diagnostics/network-check-target-xqlm2" Jul 31 18:43:46.471779 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:46.471746 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 18:43:46.471779 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:46.471768 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 18:43:46.471779 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:46.471778 2565 projected.go:196] Error preparing data for projected volume kube-api-access-2k697 for pod openshift-network-diagnostics/network-check-target-xqlm2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 18:43:46.471969 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:46.471852 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0914ed48-d402-4d77-a2e7-ca12693fd0f0-kube-api-access-2k697 podName:0914ed48-d402-4d77-a2e7-ca12693fd0f0 nodeName:}" failed. No retries permitted until 2026-07-31 18:43:47.471837645 +0000 UTC m=+4.109827426 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2k697" (UniqueName: "kubernetes.io/projected/0914ed48-d402-4d77-a2e7-ca12693fd0f0-kube-api-access-2k697") pod "network-check-target-xqlm2" (UID: "0914ed48-d402-4d77-a2e7-ca12693fd0f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 18:43:46.799922 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:46.799869 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 18:38:46 +0000 UTC" deadline="2028-01-05 08:46:06.381968111 +0000 UTC" Jul 31 18:43:46.799922 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:46.799913 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12542h2m19.582059079s" Jul 31 18:43:46.984231 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:46.984198 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf84e1304_da65_4c5c_ad8d_44e1c8b0808b.slice/crio-7c4a97b1b36b111d785747ccb9064a657f7ac40b42df0ef16a7d75d928a1b3ac WatchSource:0}: Error finding container 7c4a97b1b36b111d785747ccb9064a657f7ac40b42df0ef16a7d75d928a1b3ac: Status 404 returned error can't find the container with id 7c4a97b1b36b111d785747ccb9064a657f7ac40b42df0ef16a7d75d928a1b3ac Jul 31 18:43:46.985928 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:46.985900 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba85de76_d297_4a7b_bb30_9c12572d9811.slice/crio-232c453816b58b74dd831566c6f5533541d50dc288776a3da055c47e2e10510b WatchSource:0}: Error finding container 232c453816b58b74dd831566c6f5533541d50dc288776a3da055c47e2e10510b: Status 404 returned error can't find the container with id 232c453816b58b74dd831566c6f5533541d50dc288776a3da055c47e2e10510b Jul 31 18:43:46.986984 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:46.986955 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdddb050_55be_48e1_9a8e_df26711e4416.slice/crio-efbf124dc080a3fe1a7078d8e6865816fc331fdb3314be71b4dd7834923a94f7 WatchSource:0}: Error finding container efbf124dc080a3fe1a7078d8e6865816fc331fdb3314be71b4dd7834923a94f7: Status 404 returned error can't find the container with id efbf124dc080a3fe1a7078d8e6865816fc331fdb3314be71b4dd7834923a94f7 Jul 31 18:43:46.991651 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:46.991628 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38da7ba5_0ebc_441e_89e3_c265c2da036f.slice/crio-a96155efef804e0fad87368ea4dbf0000d5d09659ae129cb2cc38919eeb0616a WatchSource:0}: Error finding container a96155efef804e0fad87368ea4dbf0000d5d09659ae129cb2cc38919eeb0616a: Status 404 returned error can't find the container with id a96155efef804e0fad87368ea4dbf0000d5d09659ae129cb2cc38919eeb0616a Jul 31 18:43:46.994281 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:46.993493 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffecd04e_7d64_429f_95a8_20647557526d.slice/crio-af60e8843bd61802d16d825d925aad6d9b0ed0517cbcec6146bfaea212d6bc62 WatchSource:0}: Error finding container af60e8843bd61802d16d825d925aad6d9b0ed0517cbcec6146bfaea212d6bc62: Status 404 returned error can't find the container with id af60e8843bd61802d16d825d925aad6d9b0ed0517cbcec6146bfaea212d6bc62 Jul 31 18:43:46.995157 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:43:46.994543 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf200fa12_5e9c_4942_886e_cecbc15eaa00.slice/crio-c79cc2bedabe4b6ac46dfb92990726e74178139a45133e01dbd385e9510a4275 WatchSource:0}: Error finding container c79cc2bedabe4b6ac46dfb92990726e74178139a45133e01dbd385e9510a4275: Status 404 returned error can't find the container with id c79cc2bedabe4b6ac46dfb92990726e74178139a45133e01dbd385e9510a4275 Jul 31 18:43:47.379954 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:47.379553 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8223854-5549-4701-b1ce-506ed19d8abe-original-pull-secret\") pod \"global-pull-secret-syncer-2rkhw\" (UID: \"e8223854-5549-4701-b1ce-506ed19d8abe\") " pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:43:47.380124 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:47.379732 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 18:43:47.380124 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:47.379972 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee8aa68f-72ff-4118-ad38-4da03b305feb-metrics-certs\") pod \"network-metrics-daemon-h8n5q\" (UID: \"ee8aa68f-72ff-4118-ad38-4da03b305feb\") " pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:43:47.380124 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:47.380043 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8223854-5549-4701-b1ce-506ed19d8abe-original-pull-secret podName:e8223854-5549-4701-b1ce-506ed19d8abe nodeName:}" failed. No retries permitted until 2026-07-31 18:43:49.380016423 +0000 UTC m=+6.018006226 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e8223854-5549-4701-b1ce-506ed19d8abe-original-pull-secret") pod "global-pull-secret-syncer-2rkhw" (UID: "e8223854-5549-4701-b1ce-506ed19d8abe") : object "kube-system"/"original-pull-secret" not registered Jul 31 18:43:47.380124 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:47.380112 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 18:43:47.380463 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:47.380168 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee8aa68f-72ff-4118-ad38-4da03b305feb-metrics-certs podName:ee8aa68f-72ff-4118-ad38-4da03b305feb nodeName:}" failed. No retries permitted until 2026-07-31 18:43:49.380150362 +0000 UTC m=+6.018140142 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee8aa68f-72ff-4118-ad38-4da03b305feb-metrics-certs") pod "network-metrics-daemon-h8n5q" (UID: "ee8aa68f-72ff-4118-ad38-4da03b305feb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 18:43:47.480778 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:47.480738 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k697\" (UniqueName: \"kubernetes.io/projected/0914ed48-d402-4d77-a2e7-ca12693fd0f0-kube-api-access-2k697\") pod \"network-check-target-xqlm2\" (UID: \"0914ed48-d402-4d77-a2e7-ca12693fd0f0\") " pod="openshift-network-diagnostics/network-check-target-xqlm2" Jul 31 18:43:47.480948 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:47.480916 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 18:43:47.480948 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:47.480939 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 18:43:47.481084 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:47.480953 2565 projected.go:196] Error preparing data for projected volume kube-api-access-2k697 for pod openshift-network-diagnostics/network-check-target-xqlm2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 18:43:47.481084 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:47.481019 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0914ed48-d402-4d77-a2e7-ca12693fd0f0-kube-api-access-2k697 podName:0914ed48-d402-4d77-a2e7-ca12693fd0f0 nodeName:}" failed. No retries permitted until 2026-07-31 18:43:49.480998589 +0000 UTC m=+6.118988394 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2k697" (UniqueName: "kubernetes.io/projected/0914ed48-d402-4d77-a2e7-ca12693fd0f0-kube-api-access-2k697") pod "network-check-target-xqlm2" (UID: "0914ed48-d402-4d77-a2e7-ca12693fd0f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 18:43:47.800811 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:47.800740 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 18:38:46 +0000 UTC" deadline="2028-05-06 16:08:34.472699024 +0000 UTC" Jul 31 18:43:47.800811 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:47.800810 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15477h24m46.671893999s" Jul 31 18:43:47.884215 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:47.880811 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:43:47.884215 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:47.880998 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8n5q" podUID="ee8aa68f-72ff-4118-ad38-4da03b305feb" Jul 31 18:43:47.884215 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:47.881453 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:43:47.884215 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:47.881557 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rkhw" podUID="e8223854-5549-4701-b1ce-506ed19d8abe" Jul 31 18:43:47.884215 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:47.881751 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqlm2" Jul 31 18:43:47.884215 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:47.881845 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqlm2" podUID="0914ed48-d402-4d77-a2e7-ca12693fd0f0" Jul 31 18:43:47.904943 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:47.904908 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vbbdc" event={"ID":"a9871869-ce7a-407a-8bf6-c1f871893ef9","Type":"ContainerStarted","Data":"865ad69cc71a2ec12f6544d64007391d99e80b4f280fecac25155fe30fa6033c"} Jul 31 18:43:47.911782 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:47.911737 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dkbrb" event={"ID":"f200fa12-5e9c-4942-886e-cecbc15eaa00","Type":"ContainerStarted","Data":"c79cc2bedabe4b6ac46dfb92990726e74178139a45133e01dbd385e9510a4275"} Jul 31 18:43:47.919197 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:47.919134 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7s5zf" event={"ID":"ffecd04e-7d64-429f-95a8-20647557526d","Type":"ContainerStarted","Data":"af60e8843bd61802d16d825d925aad6d9b0ed0517cbcec6146bfaea212d6bc62"} Jul 31 18:43:47.939674 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:47.939643 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" event={"ID":"38da7ba5-0ebc-441e-89e3-c265c2da036f","Type":"ContainerStarted","Data":"a96155efef804e0fad87368ea4dbf0000d5d09659ae129cb2cc38919eeb0616a"} Jul 31 18:43:47.950374 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:47.950341 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fjbvh" event={"ID":"ba85de76-d297-4a7b-bb30-9c12572d9811","Type":"ContainerStarted","Data":"232c453816b58b74dd831566c6f5533541d50dc288776a3da055c47e2e10510b"} Jul 31 18:43:47.955702 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:47.955672 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf2rc" event={"ID":"f84e1304-da65-4c5c-ad8d-44e1c8b0808b","Type":"ContainerStarted","Data":"7c4a97b1b36b111d785747ccb9064a657f7ac40b42df0ef16a7d75d928a1b3ac"} Jul 31 18:43:47.971633 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:47.971602 2565 generic.go:356] "Generic (PLEG): container finished" podID="6277a79a23fe911d619a284a778ea843" containerID="44a5dd494c3918b2d0d6b26f3c507633775e6cfdbb3beffb87a14e8ef3535461" exitCode=0 Jul 31 18:43:47.971760 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:47.971686 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal" event={"ID":"6277a79a23fe911d619a284a778ea843","Type":"ContainerDied","Data":"44a5dd494c3918b2d0d6b26f3c507633775e6cfdbb3beffb87a14e8ef3535461"} Jul 31 18:43:47.979133 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:47.979108 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" event={"ID":"e1eb24da-aa9d-44e8-b12f-6e674dfd589f","Type":"ContainerStarted","Data":"9354d67fd523bfb1d0f216b8069d2f53274df652e6198a42c8a2b300fdcd6b8a"} Jul 31 18:43:47.982904 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:47.982883 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sbmpv" event={"ID":"fa723eb5-7c9f-4676-aeda-50b07733ff96","Type":"ContainerStarted","Data":"f200a76495c087ed6f0e141169bf7aa97098f930e0ed60bf72d6bf206d4c3f67"} Jul 31 18:43:47.998331 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:47.998302 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p9mqb" event={"ID":"cdddb050-55be-48e1-9a8e-df26711e4416","Type":"ContainerStarted","Data":"efbf124dc080a3fe1a7078d8e6865816fc331fdb3314be71b4dd7834923a94f7"} Jul 31 18:43:48.010385 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:48.010306 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-39.ec2.internal" event={"ID":"6781281007e7ddfaa5111df980719fef","Type":"ContainerStarted","Data":"189680ca4282e93910ce64ec060e31e41ce3263c1716c11cf87f6fdef2c0d7ef"} Jul 31 18:43:48.035287 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:48.031172 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-39.ec2.internal" podStartSLOduration=3.031151398 podStartE2EDuration="3.031151398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:43:48.030302262 +0000 UTC m=+4.668292068" watchObservedRunningTime="2026-07-31 18:43:48.031151398 +0000 UTC m=+4.669141203" Jul 31 18:43:49.031335 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:49.031285 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal" event={"ID":"6277a79a23fe911d619a284a778ea843","Type":"ContainerStarted","Data":"09efb477eaac875c0e09b818c13ce2a66bfd6fa946d8d06aeca65765cfadb8d8"} Jul 31 18:43:49.060284 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:49.060177 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-39.ec2.internal" podStartSLOduration=4.060156375 podStartE2EDuration="4.060156375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:43:49.059089661 +0000 UTC m=+5.697079467" watchObservedRunningTime="2026-07-31 18:43:49.060156375 +0000 UTC m=+5.698146179" Jul 31 18:43:49.397136 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:49.396148 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8223854-5549-4701-b1ce-506ed19d8abe-original-pull-secret\") pod \"global-pull-secret-syncer-2rkhw\" (UID: \"e8223854-5549-4701-b1ce-506ed19d8abe\") " pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:43:49.397136 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:49.396242 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee8aa68f-72ff-4118-ad38-4da03b305feb-metrics-certs\") pod \"network-metrics-daemon-h8n5q\" (UID: \"ee8aa68f-72ff-4118-ad38-4da03b305feb\") " pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:43:49.397136 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:49.396417 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 18:43:49.397136 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:49.396486 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee8aa68f-72ff-4118-ad38-4da03b305feb-metrics-certs podName:ee8aa68f-72ff-4118-ad38-4da03b305feb nodeName:}" failed. No retries permitted until 2026-07-31 18:43:53.396464366 +0000 UTC m=+10.034454159 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee8aa68f-72ff-4118-ad38-4da03b305feb-metrics-certs") pod "network-metrics-daemon-h8n5q" (UID: "ee8aa68f-72ff-4118-ad38-4da03b305feb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 18:43:49.397136 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:49.396950 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 18:43:49.397136 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:49.397011 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8223854-5549-4701-b1ce-506ed19d8abe-original-pull-secret podName:e8223854-5549-4701-b1ce-506ed19d8abe nodeName:}" failed. No retries permitted until 2026-07-31 18:43:53.396993711 +0000 UTC m=+10.034983508 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e8223854-5549-4701-b1ce-506ed19d8abe-original-pull-secret") pod "global-pull-secret-syncer-2rkhw" (UID: "e8223854-5549-4701-b1ce-506ed19d8abe") : object "kube-system"/"original-pull-secret" not registered Jul 31 18:43:49.498191 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:49.497545 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k697\" (UniqueName: \"kubernetes.io/projected/0914ed48-d402-4d77-a2e7-ca12693fd0f0-kube-api-access-2k697\") pod \"network-check-target-xqlm2\" (UID: \"0914ed48-d402-4d77-a2e7-ca12693fd0f0\") " pod="openshift-network-diagnostics/network-check-target-xqlm2" Jul 31 18:43:49.498191 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:49.497731 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 18:43:49.498191 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:49.497758 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 18:43:49.498191 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:49.497772 2565 projected.go:196] Error preparing data for projected volume kube-api-access-2k697 for pod openshift-network-diagnostics/network-check-target-xqlm2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 18:43:49.498191 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:49.497832 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0914ed48-d402-4d77-a2e7-ca12693fd0f0-kube-api-access-2k697 podName:0914ed48-d402-4d77-a2e7-ca12693fd0f0 nodeName:}" failed. No retries permitted until 2026-07-31 18:43:53.497813811 +0000 UTC m=+10.135803606 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2k697" (UniqueName: "kubernetes.io/projected/0914ed48-d402-4d77-a2e7-ca12693fd0f0-kube-api-access-2k697") pod "network-check-target-xqlm2" (UID: "0914ed48-d402-4d77-a2e7-ca12693fd0f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 18:43:49.882363 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:49.881414 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:43:49.882363 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:49.881566 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8n5q" podUID="ee8aa68f-72ff-4118-ad38-4da03b305feb" Jul 31 18:43:49.882363 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:49.881990 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:43:49.882363 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:49.882094 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rkhw" podUID="e8223854-5549-4701-b1ce-506ed19d8abe" Jul 31 18:43:49.882363 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:49.882270 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqlm2" Jul 31 18:43:49.882770 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:49.882375 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqlm2" podUID="0914ed48-d402-4d77-a2e7-ca12693fd0f0" Jul 31 18:43:51.881052 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:51.881012 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:43:51.881500 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:51.881156 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rkhw" podUID="e8223854-5549-4701-b1ce-506ed19d8abe" Jul 31 18:43:51.881754 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:51.881599 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqlm2" Jul 31 18:43:51.881754 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:51.881694 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqlm2" podUID="0914ed48-d402-4d77-a2e7-ca12693fd0f0" Jul 31 18:43:51.881949 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:51.881860 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:43:51.882003 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:51.881954 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8n5q" podUID="ee8aa68f-72ff-4118-ad38-4da03b305feb" Jul 31 18:43:53.431601 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:53.430885 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee8aa68f-72ff-4118-ad38-4da03b305feb-metrics-certs\") pod \"network-metrics-daemon-h8n5q\" (UID: \"ee8aa68f-72ff-4118-ad38-4da03b305feb\") " pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:43:53.431601 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:53.430949 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8223854-5549-4701-b1ce-506ed19d8abe-original-pull-secret\") pod \"global-pull-secret-syncer-2rkhw\" (UID: \"e8223854-5549-4701-b1ce-506ed19d8abe\") " pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:43:53.431601 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:53.431083 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 18:43:53.431601 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:53.431138 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8223854-5549-4701-b1ce-506ed19d8abe-original-pull-secret podName:e8223854-5549-4701-b1ce-506ed19d8abe nodeName:}" failed. No retries permitted until 2026-07-31 18:44:01.431120179 +0000 UTC m=+18.069109960 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e8223854-5549-4701-b1ce-506ed19d8abe-original-pull-secret") pod "global-pull-secret-syncer-2rkhw" (UID: "e8223854-5549-4701-b1ce-506ed19d8abe") : object "kube-system"/"original-pull-secret" not registered Jul 31 18:43:53.431601 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:53.431532 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 18:43:53.431601 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:53.431579 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee8aa68f-72ff-4118-ad38-4da03b305feb-metrics-certs podName:ee8aa68f-72ff-4118-ad38-4da03b305feb nodeName:}" failed. No retries permitted until 2026-07-31 18:44:01.43156366 +0000 UTC m=+18.069553441 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee8aa68f-72ff-4118-ad38-4da03b305feb-metrics-certs") pod "network-metrics-daemon-h8n5q" (UID: "ee8aa68f-72ff-4118-ad38-4da03b305feb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 18:43:53.532043 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:53.531872 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k697\" (UniqueName: \"kubernetes.io/projected/0914ed48-d402-4d77-a2e7-ca12693fd0f0-kube-api-access-2k697\") pod \"network-check-target-xqlm2\" (UID: \"0914ed48-d402-4d77-a2e7-ca12693fd0f0\") " pod="openshift-network-diagnostics/network-check-target-xqlm2" Jul 31 18:43:53.532043 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:53.532042 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 18:43:53.532281 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:53.532062 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 18:43:53.532281 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:53.532072 2565 projected.go:196] Error preparing data for projected volume kube-api-access-2k697 for pod openshift-network-diagnostics/network-check-target-xqlm2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 18:43:53.532281 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:53.532130 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0914ed48-d402-4d77-a2e7-ca12693fd0f0-kube-api-access-2k697 podName:0914ed48-d402-4d77-a2e7-ca12693fd0f0 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:01.532111578 +0000 UTC m=+18.170101374 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2k697" (UniqueName: "kubernetes.io/projected/0914ed48-d402-4d77-a2e7-ca12693fd0f0-kube-api-access-2k697") pod "network-check-target-xqlm2" (UID: "0914ed48-d402-4d77-a2e7-ca12693fd0f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 18:43:53.881425 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:53.881392 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:43:53.881425 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:53.881415 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqlm2" Jul 31 18:43:53.881665 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:53.881506 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8n5q" podUID="ee8aa68f-72ff-4118-ad38-4da03b305feb" Jul 31 18:43:53.881665 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:53.881537 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:43:53.881665 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:53.881640 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rkhw" podUID="e8223854-5549-4701-b1ce-506ed19d8abe" Jul 31 18:43:53.881810 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:53.881714 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqlm2" podUID="0914ed48-d402-4d77-a2e7-ca12693fd0f0" Jul 31 18:43:55.881192 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:55.881155 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:43:55.881680 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:55.881155 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:43:55.881680 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:55.881318 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqlm2" Jul 31 18:43:55.881680 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:55.881335 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8n5q" podUID="ee8aa68f-72ff-4118-ad38-4da03b305feb" Jul 31 18:43:55.881680 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:55.881410 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqlm2" podUID="0914ed48-d402-4d77-a2e7-ca12693fd0f0" Jul 31 18:43:55.881680 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:55.881487 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rkhw" podUID="e8223854-5549-4701-b1ce-506ed19d8abe" Jul 31 18:43:57.880598 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:57.880565 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:43:57.880598 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:57.880595 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqlm2" Jul 31 18:43:57.881005 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:57.880565 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:43:57.881005 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:57.880665 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rkhw" podUID="e8223854-5549-4701-b1ce-506ed19d8abe" Jul 31 18:43:57.881005 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:57.880742 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqlm2" podUID="0914ed48-d402-4d77-a2e7-ca12693fd0f0" Jul 31 18:43:57.881005 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:57.880805 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8n5q" podUID="ee8aa68f-72ff-4118-ad38-4da03b305feb" Jul 31 18:43:59.880635 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:59.880600 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:43:59.880635 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:59.880621 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:43:59.880635 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:43:59.880628 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqlm2" Jul 31 18:43:59.881055 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:59.880743 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8n5q" podUID="ee8aa68f-72ff-4118-ad38-4da03b305feb" Jul 31 18:43:59.881055 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:59.880873 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rkhw" podUID="e8223854-5549-4701-b1ce-506ed19d8abe" Jul 31 18:43:59.881055 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:43:59.880943 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqlm2" podUID="0914ed48-d402-4d77-a2e7-ca12693fd0f0" Jul 31 18:44:01.489544 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:01.489399 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee8aa68f-72ff-4118-ad38-4da03b305feb-metrics-certs\") pod \"network-metrics-daemon-h8n5q\" (UID: \"ee8aa68f-72ff-4118-ad38-4da03b305feb\") " pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:44:01.489544 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:01.489475 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8223854-5549-4701-b1ce-506ed19d8abe-original-pull-secret\") pod \"global-pull-secret-syncer-2rkhw\" (UID: \"e8223854-5549-4701-b1ce-506ed19d8abe\") " pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:44:01.489544 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:01.489522 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 18:44:01.490105 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:01.489584 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 18:44:01.490105 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:01.489602 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee8aa68f-72ff-4118-ad38-4da03b305feb-metrics-certs podName:ee8aa68f-72ff-4118-ad38-4da03b305feb nodeName:}" failed. No retries permitted until 2026-07-31 18:44:17.489581078 +0000 UTC m=+34.127570857 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee8aa68f-72ff-4118-ad38-4da03b305feb-metrics-certs") pod "network-metrics-daemon-h8n5q" (UID: "ee8aa68f-72ff-4118-ad38-4da03b305feb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 18:44:01.490105 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:01.489629 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8223854-5549-4701-b1ce-506ed19d8abe-original-pull-secret podName:e8223854-5549-4701-b1ce-506ed19d8abe nodeName:}" failed. No retries permitted until 2026-07-31 18:44:17.489613065 +0000 UTC m=+34.127602849 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e8223854-5549-4701-b1ce-506ed19d8abe-original-pull-secret") pod "global-pull-secret-syncer-2rkhw" (UID: "e8223854-5549-4701-b1ce-506ed19d8abe") : object "kube-system"/"original-pull-secret" not registered Jul 31 18:44:01.590773 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:01.590734 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k697\" (UniqueName: \"kubernetes.io/projected/0914ed48-d402-4d77-a2e7-ca12693fd0f0-kube-api-access-2k697\") pod \"network-check-target-xqlm2\" (UID: \"0914ed48-d402-4d77-a2e7-ca12693fd0f0\") " pod="openshift-network-diagnostics/network-check-target-xqlm2" Jul 31 18:44:01.590935 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:01.590898 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 18:44:01.590935 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:01.590925 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 18:44:01.591043 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:01.590938 2565 projected.go:196] Error preparing data for projected volume kube-api-access-2k697 for pod openshift-network-diagnostics/network-check-target-xqlm2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 18:44:01.591043 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:01.591005 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0914ed48-d402-4d77-a2e7-ca12693fd0f0-kube-api-access-2k697 podName:0914ed48-d402-4d77-a2e7-ca12693fd0f0 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:17.590987913 +0000 UTC m=+34.228977704 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2k697" (UniqueName: "kubernetes.io/projected/0914ed48-d402-4d77-a2e7-ca12693fd0f0-kube-api-access-2k697") pod "network-check-target-xqlm2" (UID: "0914ed48-d402-4d77-a2e7-ca12693fd0f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 18:44:01.880978 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:01.880940 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:44:01.880978 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:01.880961 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:44:01.881213 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:01.880940 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqlm2" Jul 31 18:44:01.881213 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:01.881059 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rkhw" podUID="e8223854-5549-4701-b1ce-506ed19d8abe" Jul 31 18:44:01.881213 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:01.881132 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqlm2" podUID="0914ed48-d402-4d77-a2e7-ca12693fd0f0" Jul 31 18:44:01.881400 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:01.881221 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8n5q" podUID="ee8aa68f-72ff-4118-ad38-4da03b305feb" Jul 31 18:44:03.881611 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:03.881576 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:44:03.882101 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:03.881686 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:44:03.882101 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:03.881696 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rkhw" podUID="e8223854-5549-4701-b1ce-506ed19d8abe" Jul 31 18:44:03.882101 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:03.881821 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8n5q" podUID="ee8aa68f-72ff-4118-ad38-4da03b305feb" Jul 31 18:44:03.882101 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:03.881855 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqlm2" Jul 31 18:44:03.882101 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:03.881984 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqlm2" podUID="0914ed48-d402-4d77-a2e7-ca12693fd0f0" Jul 31 18:44:05.881523 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:05.881230 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:44:05.882258 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:05.882169 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rkhw" podUID="e8223854-5549-4701-b1ce-506ed19d8abe" Jul 31 18:44:05.882480 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:05.882372 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqlm2" Jul 31 18:44:05.882659 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:05.882573 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqlm2" podUID="0914ed48-d402-4d77-a2e7-ca12693fd0f0" Jul 31 18:44:05.882737 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:05.882643 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:44:05.882800 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:05.882759 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8n5q" podUID="ee8aa68f-72ff-4118-ad38-4da03b305feb" Jul 31 18:44:06.067441 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:06.067252 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vbbdc" event={"ID":"a9871869-ce7a-407a-8bf6-c1f871893ef9","Type":"ContainerStarted","Data":"50aaec6863ef42c7f29153b9cea80f53cc1e952208d35d90d5df65338e6c4249"} Jul 31 18:44:06.068777 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:06.068756 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7s5zf" event={"ID":"ffecd04e-7d64-429f-95a8-20647557526d","Type":"ContainerStarted","Data":"dcf81df7912ddba5453097e7e6020b80ad72ec56da4e359ca471fe93bd8ed00c"} Jul 31 18:44:06.080472 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:06.080445 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" event={"ID":"38da7ba5-0ebc-441e-89e3-c265c2da036f","Type":"ContainerStarted","Data":"4856d5c93c5791d4f44267799a064326ac8a2fdf0e740e35140b24b13b7277f0"} Jul 31 18:44:06.080577 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:06.080502 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" event={"ID":"38da7ba5-0ebc-441e-89e3-c265c2da036f","Type":"ContainerStarted","Data":"c1a7477d712ed12110a7510b91e6a1b8dc06f0b700e65fa80d5358e1c6ec271e"} Jul 31 18:44:06.080577 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:06.080539 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" event={"ID":"38da7ba5-0ebc-441e-89e3-c265c2da036f","Type":"ContainerStarted","Data":"42026965fdd6bee51c10fa4ac1d9421a4b2e8f63ab544eaf86b9a8f38ffa5c62"} Jul 31 18:44:06.081681 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:06.081662 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fjbvh" event={"ID":"ba85de76-d297-4a7b-bb30-9c12572d9811","Type":"ContainerStarted","Data":"d5214d14158b59dba5853f877bac0e9bb351d4d34759d47d9112900b12c472ac"} Jul 31 18:44:06.082882 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:06.082861 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf2rc" event={"ID":"f84e1304-da65-4c5c-ad8d-44e1c8b0808b","Type":"ContainerStarted","Data":"715b96a69cd353796b3f0deebc28add2ba8aa7b12e0e4bc6dfc633db2ba118b3"} Jul 31 18:44:06.083966 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:06.083949 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" event={"ID":"e1eb24da-aa9d-44e8-b12f-6e674dfd589f","Type":"ContainerStarted","Data":"c55ea4747c8be0810d3320103429f45e81dd1b720473005c4ccfdc6bf10662f9"} Jul 31 18:44:06.085155 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:06.085133 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sbmpv" event={"ID":"fa723eb5-7c9f-4676-aeda-50b07733ff96","Type":"ContainerStarted","Data":"9bb5b8cc5ca8eb24bef123236dcade92d9812ada2ee84fda4b1dbc1406416448"} Jul 31 18:44:06.086539 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:06.086514 2565 generic.go:356] "Generic (PLEG): container finished" podID="cdddb050-55be-48e1-9a8e-df26711e4416" containerID="54390b646079fde88971efa4afd95e0bc3a4981bf415f9b1e8bd701d753b3a9c" exitCode=0 Jul 31 18:44:06.086631 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:06.086546 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p9mqb" event={"ID":"cdddb050-55be-48e1-9a8e-df26711e4416","Type":"ContainerDied","Data":"54390b646079fde88971efa4afd95e0bc3a4981bf415f9b1e8bd701d753b3a9c"} Jul 31 18:44:06.088480 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:06.088442 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vbbdc" podStartSLOduration=3.357190353 podStartE2EDuration="22.088431312s" totalImagesPullingTime="18.731240959s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:06.088201176 +0000 UTC m=+22.726190976" watchObservedRunningTime="2026-07-31 18:44:06.088431312 +0000 UTC m=+22.726421112" Jul 31 18:44:06.106870 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:06.106832 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7s5zf" podStartSLOduration=7.434454979 podStartE2EDuration="22.106821331s" totalImagesPullingTime="14.672366352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:06.106173947 +0000 UTC m=+22.744163746" watchObservedRunningTime="2026-07-31 18:44:06.106821331 +0000 UTC m=+22.744811130" Jul 31 18:44:06.127013 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:06.126978 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jh7cb" podStartSLOduration=3.432869014 podStartE2EDuration="22.126966942s" totalImagesPullingTime="18.694097928s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:06.126483349 +0000 UTC m=+22.764473143" watchObservedRunningTime="2026-07-31 18:44:06.126966942 +0000 UTC m=+22.764956739" Jul 31 18:44:06.144134 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:06.144101 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-sbmpv" podStartSLOduration=3.467552667 podStartE2EDuration="22.144089642s" totalImagesPullingTime="18.676536975s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:06.14381704 +0000 UTC m=+22.781806839" watchObservedRunningTime="2026-07-31 18:44:06.144089642 +0000 UTC m=+22.782079441" Jul 31 18:44:06.185344 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:06.185306 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fjbvh" podStartSLOduration=3.506414533 podStartE2EDuration="22.18529369s" totalImagesPullingTime="18.678879157s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:06.185215127 +0000 UTC m=+22.823204928" watchObservedRunningTime="2026-07-31 18:44:06.18529369 +0000 UTC m=+22.823283490" Jul 31 18:44:07.092732 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:07.092701 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" event={"ID":"38da7ba5-0ebc-441e-89e3-c265c2da036f","Type":"ContainerStarted","Data":"b47c81517fef598a62de1a3a46c6d4b706f777933ce9f222436ef75b2b58af9b"} Jul 31 18:44:07.093406 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:07.092824 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" event={"ID":"38da7ba5-0ebc-441e-89e3-c265c2da036f","Type":"ContainerStarted","Data":"417a403c29c8b8eeb2416eb96a00993b5a1f153068c2084130f8edd6c88096bb"} Jul 31 18:44:07.093406 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:07.092875 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" event={"ID":"38da7ba5-0ebc-441e-89e3-c265c2da036f","Type":"ContainerStarted","Data":"2e08f2f842b7c0c0387fa12401a9e28abc4a729f200a1500eb1270ac30427969"} Jul 31 18:44:07.206379 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:07.206352 2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 31 18:44:07.817047 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:07.816931 2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-31T18:44:07.206373921Z","UUID":"ebfca496-5f9a-41f0-adcb-35ebf53d94bc","Handler":null,"Name":"","Endpoint":""} Jul 31 18:44:07.818325 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:07.818300 2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 31 18:44:07.818451 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:07.818344 2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 31 18:44:07.880141 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:07.880113 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:44:07.880363 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:07.880271 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8n5q" podUID="ee8aa68f-72ff-4118-ad38-4da03b305feb" Jul 31 18:44:07.880363 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:07.880296 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqlm2" Jul 31 18:44:07.880496 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:07.880381 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqlm2" podUID="0914ed48-d402-4d77-a2e7-ca12693fd0f0" Jul 31 18:44:07.880496 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:07.880113 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:44:07.880496 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:07.880478 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rkhw" podUID="e8223854-5549-4701-b1ce-506ed19d8abe" Jul 31 18:44:08.096946 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:08.096722 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dkbrb" event={"ID":"f200fa12-5e9c-4942-886e-cecbc15eaa00","Type":"ContainerStarted","Data":"0e3b9259c178437776e17a5629f54cd4f28538af73f7d2dc901208ad566eb63d"} Jul 31 18:44:08.098878 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:08.098849 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf2rc" event={"ID":"f84e1304-da65-4c5c-ad8d-44e1c8b0808b","Type":"ContainerStarted","Data":"eb91205a41fbeda1e0d72c3e40aaf5dbb4431c90e65468240cf3d9092fcba19b"} Jul 31 18:44:08.113607 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:08.113556 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dkbrb" podStartSLOduration=5.442818949 podStartE2EDuration="24.11354355s" totalImagesPullingTime="18.670724601s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:08.113069624 +0000 UTC m=+24.751059428" watchObservedRunningTime="2026-07-31 18:44:08.11354355 +0000 UTC m=+24.751533350" Jul 31 18:44:09.104588 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:09.104545 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" event={"ID":"38da7ba5-0ebc-441e-89e3-c265c2da036f","Type":"ContainerStarted","Data":"f64cfbe70b7560e7067c83963ed320e037aa2cdb379f5fdf792f6117f4f1dd64"} Jul 31 18:44:09.108555 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:09.108528 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf2rc" event={"ID":"f84e1304-da65-4c5c-ad8d-44e1c8b0808b","Type":"ContainerStarted","Data":"c9f3df3828628a844ad8f54dd70e7e4d254af0ce6221a8b964b55ff21330366b"} Jul 31 18:44:09.129209 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:09.129170 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pf2rc" podStartSLOduration=4.241153787 podStartE2EDuration="25.129157669s" totalImagesPullingTime="20.888003882s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:43:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:09.128858062 +0000 UTC m=+25.766847864" watchObservedRunningTime="2026-07-31 18:44:09.129157669 +0000 UTC m=+25.767147469" Jul 31 18:44:09.416574 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:09.416501 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7s5zf" Jul 31 18:44:09.417041 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:09.417022 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7s5zf" Jul 31 18:44:09.880468 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:09.880436 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:44:09.880660 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:09.880477 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqlm2" Jul 31 18:44:09.880660 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:09.880436 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:44:09.880660 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:09.880584 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rkhw" podUID="e8223854-5549-4701-b1ce-506ed19d8abe" Jul 31 18:44:09.880804 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:09.880723 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8n5q" podUID="ee8aa68f-72ff-4118-ad38-4da03b305feb" Jul 31 18:44:09.880902 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:09.880862 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqlm2" podUID="0914ed48-d402-4d77-a2e7-ca12693fd0f0" Jul 31 18:44:10.111425 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:10.111401 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7s5zf" Jul 31 18:44:10.111863 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:10.111660 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7s5zf" Jul 31 18:44:11.116715 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:11.116554 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" event={"ID":"38da7ba5-0ebc-441e-89e3-c265c2da036f","Type":"ContainerStarted","Data":"6e0f55cc26370a2e0e71bdd24c0e8cd2e97bbe39450b005d805b2d071dfc7154"} Jul 31 18:44:11.147322 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:11.147070 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" podStartSLOduration=8.398434098 podStartE2EDuration="27.147047953s" totalImagesPullingTime="18.748613855s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:11.14563645 +0000 UTC m=+27.783626263" watchObservedRunningTime="2026-07-31 18:44:11.147047953 +0000 UTC m=+27.785037754" Jul 31 18:44:11.880939 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:11.880684 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:44:11.881112 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:11.880738 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqlm2" Jul 31 18:44:11.881112 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:11.881045 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rkhw" podUID="e8223854-5549-4701-b1ce-506ed19d8abe" Jul 31 18:44:11.881112 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:11.880802 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:44:11.881112 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:11.881096 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqlm2" podUID="0914ed48-d402-4d77-a2e7-ca12693fd0f0" Jul 31 18:44:11.881261 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:11.881170 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8n5q" podUID="ee8aa68f-72ff-4118-ad38-4da03b305feb" Jul 31 18:44:12.120821 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:12.120787 2565 generic.go:356] "Generic (PLEG): container finished" podID="cdddb050-55be-48e1-9a8e-df26711e4416" containerID="7557d278664b90f020f87490c3ddbdef529792b4018b4f341b25d88b20d1504d" exitCode=0 Jul 31 18:44:12.121451 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:12.120867 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p9mqb" event={"ID":"cdddb050-55be-48e1-9a8e-df26711e4416","Type":"ContainerDied","Data":"7557d278664b90f020f87490c3ddbdef529792b4018b4f341b25d88b20d1504d"} Jul 31 18:44:12.121517 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:12.121449 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:44:12.121517 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:12.121515 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:44:12.121604 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:12.121555 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:44:12.136988 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:12.136886 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:44:12.137116 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:12.137018 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:44:13.224410 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:13.224375 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h8n5q"] Jul 31 18:44:13.224966 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:13.224494 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:44:13.224966 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:13.224615 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8n5q" podUID="ee8aa68f-72ff-4118-ad38-4da03b305feb" Jul 31 18:44:13.227182 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:13.227160 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2rkhw"] Jul 31 18:44:13.227311 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:13.227260 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:44:13.227377 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:13.227354 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rkhw" podUID="e8223854-5549-4701-b1ce-506ed19d8abe" Jul 31 18:44:13.229840 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:13.229815 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xqlm2"] Jul 31 18:44:13.229922 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:13.229890 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqlm2" Jul 31 18:44:13.229982 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:13.229965 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqlm2" podUID="0914ed48-d402-4d77-a2e7-ca12693fd0f0" Jul 31 18:44:14.127568 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:14.127537 2565 generic.go:356] "Generic (PLEG): container finished" podID="cdddb050-55be-48e1-9a8e-df26711e4416" containerID="3ce5df8242921f0ff5dce053d68af482eaad482e7070ea353221bb0be7262b8f" exitCode=0 Jul 31 18:44:14.127721 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:14.127620 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p9mqb" event={"ID":"cdddb050-55be-48e1-9a8e-df26711e4416","Type":"ContainerDied","Data":"3ce5df8242921f0ff5dce053d68af482eaad482e7070ea353221bb0be7262b8f"} Jul 31 18:44:14.880537 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:14.880508 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqlm2" Jul 31 18:44:14.880537 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:14.880526 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:44:14.881065 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:14.880520 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:44:14.881065 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:14.880637 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqlm2" podUID="0914ed48-d402-4d77-a2e7-ca12693fd0f0" Jul 31 18:44:14.881065 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:14.880724 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8n5q" podUID="ee8aa68f-72ff-4118-ad38-4da03b305feb" Jul 31 18:44:14.881065 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:14.880811 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rkhw" podUID="e8223854-5549-4701-b1ce-506ed19d8abe" Jul 31 18:44:16.134784 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:16.134603 2565 generic.go:356] "Generic (PLEG): container finished" podID="cdddb050-55be-48e1-9a8e-df26711e4416" containerID="2d704d02eebc2e8793b7be76a80d488aed2320dc60a97d6ff7db98b923d45048" exitCode=0 Jul 31 18:44:16.135254 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:16.134685 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p9mqb" event={"ID":"cdddb050-55be-48e1-9a8e-df26711e4416","Type":"ContainerDied","Data":"2d704d02eebc2e8793b7be76a80d488aed2320dc60a97d6ff7db98b923d45048"} Jul 31 18:44:16.881081 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:16.881042 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:44:16.881288 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:16.881148 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqlm2" Jul 31 18:44:16.881288 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:16.881152 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rkhw" podUID="e8223854-5549-4701-b1ce-506ed19d8abe" Jul 31 18:44:16.881288 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:16.881196 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:44:16.881456 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:16.881300 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqlm2" podUID="0914ed48-d402-4d77-a2e7-ca12693fd0f0" Jul 31 18:44:16.881456 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:16.881400 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8n5q" podUID="ee8aa68f-72ff-4118-ad38-4da03b305feb" Jul 31 18:44:17.510886 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:17.510851 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee8aa68f-72ff-4118-ad38-4da03b305feb-metrics-certs\") pod \"network-metrics-daemon-h8n5q\" (UID: \"ee8aa68f-72ff-4118-ad38-4da03b305feb\") " pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:44:17.511336 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:17.510898 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8223854-5549-4701-b1ce-506ed19d8abe-original-pull-secret\") pod \"global-pull-secret-syncer-2rkhw\" (UID: \"e8223854-5549-4701-b1ce-506ed19d8abe\") " pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:44:17.511336 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:17.511017 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 18:44:17.511336 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:17.511017 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 18:44:17.511336 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:17.511074 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8223854-5549-4701-b1ce-506ed19d8abe-original-pull-secret podName:e8223854-5549-4701-b1ce-506ed19d8abe nodeName:}" failed. No retries permitted until 2026-07-31 18:44:49.511058322 +0000 UTC m=+66.149048100 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e8223854-5549-4701-b1ce-506ed19d8abe-original-pull-secret") pod "global-pull-secret-syncer-2rkhw" (UID: "e8223854-5549-4701-b1ce-506ed19d8abe") : object "kube-system"/"original-pull-secret" not registered Jul 31 18:44:17.511336 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:17.511094 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee8aa68f-72ff-4118-ad38-4da03b305feb-metrics-certs podName:ee8aa68f-72ff-4118-ad38-4da03b305feb nodeName:}" failed. No retries permitted until 2026-07-31 18:44:49.511084159 +0000 UTC m=+66.149073939 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee8aa68f-72ff-4118-ad38-4da03b305feb-metrics-certs") pod "network-metrics-daemon-h8n5q" (UID: "ee8aa68f-72ff-4118-ad38-4da03b305feb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 18:44:17.611642 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:17.611607 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k697\" (UniqueName: \"kubernetes.io/projected/0914ed48-d402-4d77-a2e7-ca12693fd0f0-kube-api-access-2k697\") pod \"network-check-target-xqlm2\" (UID: \"0914ed48-d402-4d77-a2e7-ca12693fd0f0\") " pod="openshift-network-diagnostics/network-check-target-xqlm2" Jul 31 18:44:17.611801 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:17.611740 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 18:44:17.611801 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:17.611755 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 18:44:17.611801 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:17.611764 2565 projected.go:196] Error preparing data for projected volume kube-api-access-2k697 for pod openshift-network-diagnostics/network-check-target-xqlm2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 18:44:17.611944 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:17.611811 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0914ed48-d402-4d77-a2e7-ca12693fd0f0-kube-api-access-2k697 podName:0914ed48-d402-4d77-a2e7-ca12693fd0f0 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:49.611798437 +0000 UTC m=+66.249788215 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2k697" (UniqueName: "kubernetes.io/projected/0914ed48-d402-4d77-a2e7-ca12693fd0f0-kube-api-access-2k697") pod "network-check-target-xqlm2" (UID: "0914ed48-d402-4d77-a2e7-ca12693fd0f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 18:44:18.470982 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:18.470897 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:18.663230 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:18.663195 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:18.742623 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:18.742550 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-39.ec2.internal" event="NodeReady" Jul 31 18:44:18.742777 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:18.742711 2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 31 18:44:18.830386 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:18.830346 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9kv6f"] Jul 31 18:44:18.833957 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:18.833932 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9kv6f" Jul 31 18:44:18.836471 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:18.836449 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-96c4f\"" Jul 31 18:44:18.836592 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:18.836471 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 31 18:44:18.836592 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:18.836452 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 31 18:44:18.836989 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:18.836965 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 31 18:44:18.851034 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:18.851007 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9kv6f"] Jul 31 18:44:18.874114 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:18.874085 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dgmn6"] Jul 31 18:44:18.877906 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:18.877885 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dgmn6" Jul 31 18:44:18.880374 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:18.880141 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:44:18.880374 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:18.880327 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:44:18.880374 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:18.880353 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqlm2" Jul 31 18:44:18.880822 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:18.880800 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 31 18:44:18.880907 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:18.880806 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 31 18:44:18.880955 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:18.880919 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v8wjd\"" Jul 31 18:44:18.882481 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:18.882451 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s4mbt\"" Jul 31 18:44:18.882590 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:18.882524 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 31 18:44:18.883855 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:18.883836 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 31 18:44:18.884850 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:18.884761 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 31 18:44:18.884850 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:18.884814 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 31 18:44:18.884850 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:18.884840 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-99tgj\"" Jul 31 18:44:18.891746 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:18.891723 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dgmn6"] Jul 31 18:44:18.919421 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:18.919393 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9bd58afe-8ba9-4f95-ae27-c97586229a7a-cert\") pod \"ingress-canary-9kv6f\" (UID: \"9bd58afe-8ba9-4f95-ae27-c97586229a7a\") " pod="openshift-ingress-canary/ingress-canary-9kv6f" Jul 31 18:44:18.919555 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:18.919438 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c675z\" (UniqueName: \"kubernetes.io/projected/9bd58afe-8ba9-4f95-ae27-c97586229a7a-kube-api-access-c675z\") pod \"ingress-canary-9kv6f\" (UID: \"9bd58afe-8ba9-4f95-ae27-c97586229a7a\") " pod="openshift-ingress-canary/ingress-canary-9kv6f" Jul 31 18:44:19.020100 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:19.020061 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p98s2\" (UniqueName: \"kubernetes.io/projected/98250d35-1d7d-4f70-8350-b7865456ae46-kube-api-access-p98s2\") pod \"dns-default-dgmn6\" (UID: \"98250d35-1d7d-4f70-8350-b7865456ae46\") " pod="openshift-dns/dns-default-dgmn6" Jul 31 18:44:19.020308 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:19.020108 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98250d35-1d7d-4f70-8350-b7865456ae46-metrics-tls\") pod \"dns-default-dgmn6\" (UID: \"98250d35-1d7d-4f70-8350-b7865456ae46\") " pod="openshift-dns/dns-default-dgmn6" Jul 31 18:44:19.020308 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:19.020196 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9bd58afe-8ba9-4f95-ae27-c97586229a7a-cert\") pod \"ingress-canary-9kv6f\" (UID: \"9bd58afe-8ba9-4f95-ae27-c97586229a7a\") " pod="openshift-ingress-canary/ingress-canary-9kv6f" Jul 31 18:44:19.020308 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:19.020274 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c675z\" (UniqueName: \"kubernetes.io/projected/9bd58afe-8ba9-4f95-ae27-c97586229a7a-kube-api-access-c675z\") pod \"ingress-canary-9kv6f\" (UID: \"9bd58afe-8ba9-4f95-ae27-c97586229a7a\") " pod="openshift-ingress-canary/ingress-canary-9kv6f" Jul 31 18:44:19.020308 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:19.020300 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/98250d35-1d7d-4f70-8350-b7865456ae46-config-volume\") pod \"dns-default-dgmn6\" (UID: \"98250d35-1d7d-4f70-8350-b7865456ae46\") " pod="openshift-dns/dns-default-dgmn6" Jul 31 18:44:19.020525 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:19.020363 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98250d35-1d7d-4f70-8350-b7865456ae46-tmp-dir\") pod \"dns-default-dgmn6\" (UID: \"98250d35-1d7d-4f70-8350-b7865456ae46\") " pod="openshift-dns/dns-default-dgmn6" Jul 31 18:44:19.025100 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:19.025071 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9bd58afe-8ba9-4f95-ae27-c97586229a7a-cert\") pod \"ingress-canary-9kv6f\" (UID: \"9bd58afe-8ba9-4f95-ae27-c97586229a7a\") " pod="openshift-ingress-canary/ingress-canary-9kv6f" Jul 31 18:44:19.034622 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:19.034597 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c675z\" (UniqueName: \"kubernetes.io/projected/9bd58afe-8ba9-4f95-ae27-c97586229a7a-kube-api-access-c675z\") pod \"ingress-canary-9kv6f\" (UID: \"9bd58afe-8ba9-4f95-ae27-c97586229a7a\") " pod="openshift-ingress-canary/ingress-canary-9kv6f" Jul 31 18:44:19.121581 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:19.121542 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98250d35-1d7d-4f70-8350-b7865456ae46-tmp-dir\") pod \"dns-default-dgmn6\" (UID: \"98250d35-1d7d-4f70-8350-b7865456ae46\") " pod="openshift-dns/dns-default-dgmn6" Jul 31 18:44:19.121768 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:19.121605 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p98s2\" (UniqueName: \"kubernetes.io/projected/98250d35-1d7d-4f70-8350-b7865456ae46-kube-api-access-p98s2\") pod \"dns-default-dgmn6\" (UID: \"98250d35-1d7d-4f70-8350-b7865456ae46\") " pod="openshift-dns/dns-default-dgmn6" Jul 31 18:44:19.121768 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:19.121632 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98250d35-1d7d-4f70-8350-b7865456ae46-metrics-tls\") pod \"dns-default-dgmn6\" (UID: \"98250d35-1d7d-4f70-8350-b7865456ae46\") " pod="openshift-dns/dns-default-dgmn6" Jul 31 18:44:19.121768 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:19.121685 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/98250d35-1d7d-4f70-8350-b7865456ae46-config-volume\") pod \"dns-default-dgmn6\" (UID: \"98250d35-1d7d-4f70-8350-b7865456ae46\") " pod="openshift-dns/dns-default-dgmn6" Jul 31 18:44:19.122040 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:19.121941 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98250d35-1d7d-4f70-8350-b7865456ae46-tmp-dir\") pod \"dns-default-dgmn6\" (UID: \"98250d35-1d7d-4f70-8350-b7865456ae46\") " pod="openshift-dns/dns-default-dgmn6" Jul 31 18:44:19.122076 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:19.122044 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 18:44:19.122125 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:19.122112 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98250d35-1d7d-4f70-8350-b7865456ae46-metrics-tls podName:98250d35-1d7d-4f70-8350-b7865456ae46 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:19.622095056 +0000 UTC m=+36.260084840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98250d35-1d7d-4f70-8350-b7865456ae46-metrics-tls") pod "dns-default-dgmn6" (UID: "98250d35-1d7d-4f70-8350-b7865456ae46") : secret "dns-default-metrics-tls" not found Jul 31 18:44:19.122283 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:19.122234 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/98250d35-1d7d-4f70-8350-b7865456ae46-config-volume\") pod \"dns-default-dgmn6\" (UID: \"98250d35-1d7d-4f70-8350-b7865456ae46\") " pod="openshift-dns/dns-default-dgmn6" Jul 31 18:44:19.136219 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:19.136185 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p98s2\" (UniqueName: \"kubernetes.io/projected/98250d35-1d7d-4f70-8350-b7865456ae46-kube-api-access-p98s2\") pod \"dns-default-dgmn6\" (UID: \"98250d35-1d7d-4f70-8350-b7865456ae46\") " pod="openshift-dns/dns-default-dgmn6" Jul 31 18:44:19.144887 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:19.144858 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9kv6f" Jul 31 18:44:19.306648 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:19.306359 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9kv6f"] Jul 31 18:44:19.312796 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:44:19.312766 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd58afe_8ba9_4f95_ae27_c97586229a7a.slice/crio-6536e694b931e01aeea142eab99e90efd534dcddaa9c015fd734f5f001d50870 WatchSource:0}: Error finding container 6536e694b931e01aeea142eab99e90efd534dcddaa9c015fd734f5f001d50870: Status 404 returned error can't find the container with id 6536e694b931e01aeea142eab99e90efd534dcddaa9c015fd734f5f001d50870 Jul 31 18:44:19.626551 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:19.626463 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98250d35-1d7d-4f70-8350-b7865456ae46-metrics-tls\") pod \"dns-default-dgmn6\" (UID: \"98250d35-1d7d-4f70-8350-b7865456ae46\") " pod="openshift-dns/dns-default-dgmn6" Jul 31 18:44:19.626728 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:19.626629 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 18:44:19.626728 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:19.626687 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98250d35-1d7d-4f70-8350-b7865456ae46-metrics-tls podName:98250d35-1d7d-4f70-8350-b7865456ae46 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:20.626672709 +0000 UTC m=+37.264662486 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98250d35-1d7d-4f70-8350-b7865456ae46-metrics-tls") pod "dns-default-dgmn6" (UID: "98250d35-1d7d-4f70-8350-b7865456ae46") : secret "dns-default-metrics-tls" not found Jul 31 18:44:20.146080 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:20.146039 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9kv6f" event={"ID":"9bd58afe-8ba9-4f95-ae27-c97586229a7a","Type":"ContainerStarted","Data":"6536e694b931e01aeea142eab99e90efd534dcddaa9c015fd734f5f001d50870"} Jul 31 18:44:20.636556 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:20.636516 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98250d35-1d7d-4f70-8350-b7865456ae46-metrics-tls\") pod \"dns-default-dgmn6\" (UID: \"98250d35-1d7d-4f70-8350-b7865456ae46\") " pod="openshift-dns/dns-default-dgmn6" Jul 31 18:44:20.636754 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:20.636699 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 18:44:20.636808 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:20.636772 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98250d35-1d7d-4f70-8350-b7865456ae46-metrics-tls podName:98250d35-1d7d-4f70-8350-b7865456ae46 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:22.636756104 +0000 UTC m=+39.274745900 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98250d35-1d7d-4f70-8350-b7865456ae46-metrics-tls") pod "dns-default-dgmn6" (UID: "98250d35-1d7d-4f70-8350-b7865456ae46") : secret "dns-default-metrics-tls" not found Jul 31 18:44:21.151796 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:21.151709 2565 generic.go:356] "Generic (PLEG): container finished" podID="cdddb050-55be-48e1-9a8e-df26711e4416" containerID="1848f3eec6ffa65227954f3d1583086828785bf32c907d9eb00658a9a4276742" exitCode=0 Jul 31 18:44:21.151796 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:21.151771 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p9mqb" event={"ID":"cdddb050-55be-48e1-9a8e-df26711e4416","Type":"ContainerDied","Data":"1848f3eec6ffa65227954f3d1583086828785bf32c907d9eb00658a9a4276742"} Jul 31 18:44:21.928029 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:21.927989 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:22.050154 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:22.050102 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:22.157424 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:22.157389 2565 generic.go:356] "Generic (PLEG): container finished" podID="cdddb050-55be-48e1-9a8e-df26711e4416" containerID="916de787871428d919bbd29a679db8bae54dc218c1b6f8382f81ad93b1170d4f" exitCode=0 Jul 31 18:44:22.157424 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:22.157429 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p9mqb" event={"ID":"cdddb050-55be-48e1-9a8e-df26711e4416","Type":"ContainerDied","Data":"916de787871428d919bbd29a679db8bae54dc218c1b6f8382f81ad93b1170d4f"} Jul 31 18:44:22.653427 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:22.653392 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98250d35-1d7d-4f70-8350-b7865456ae46-metrics-tls\") pod \"dns-default-dgmn6\" (UID: \"98250d35-1d7d-4f70-8350-b7865456ae46\") " pod="openshift-dns/dns-default-dgmn6" Jul 31 18:44:22.653550 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:22.653508 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 18:44:22.653613 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:22.653600 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98250d35-1d7d-4f70-8350-b7865456ae46-metrics-tls podName:98250d35-1d7d-4f70-8350-b7865456ae46 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:26.653568439 +0000 UTC m=+43.291558220 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98250d35-1d7d-4f70-8350-b7865456ae46-metrics-tls") pod "dns-default-dgmn6" (UID: "98250d35-1d7d-4f70-8350-b7865456ae46") : secret "dns-default-metrics-tls" not found Jul 31 18:44:23.163666 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:23.163412 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p9mqb" event={"ID":"cdddb050-55be-48e1-9a8e-df26711e4416","Type":"ContainerStarted","Data":"72c1e0ff19c12bf80dded3d3c61335636486ff22a8e6abc7d4ea1b6bb6dd22ea"} Jul 31 18:44:23.165281 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:23.165251 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9kv6f" event={"ID":"9bd58afe-8ba9-4f95-ae27-c97586229a7a","Type":"ContainerStarted","Data":"c18ea6a74b9661aa4e11aac90c6366f31bfd68b739422519833fa2015e4af5e9"} Jul 31 18:44:23.211878 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:23.211785 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-p9mqb" podStartSLOduration=8.22710422 podStartE2EDuration="39.211768401s" totalImagesPullingTime="30.984664181s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:43:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:23.210497158 +0000 UTC m=+39.848486980" watchObservedRunningTime="2026-07-31 18:44:23.211768401 +0000 UTC m=+39.849758202" Jul 31 18:44:23.239549 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:23.239520 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:23.354202 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:23.354168 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:23.441657 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:23.441627 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:24.548144 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:24.548107 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:24.652260 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:24.652213 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:26.681542 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:26.681504 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98250d35-1d7d-4f70-8350-b7865456ae46-metrics-tls\") pod \"dns-default-dgmn6\" (UID: \"98250d35-1d7d-4f70-8350-b7865456ae46\") " pod="openshift-dns/dns-default-dgmn6" Jul 31 18:44:26.681921 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:26.681667 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 18:44:26.681921 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:26.681735 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98250d35-1d7d-4f70-8350-b7865456ae46-metrics-tls podName:98250d35-1d7d-4f70-8350-b7865456ae46 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:34.68172004 +0000 UTC m=+51.319709818 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98250d35-1d7d-4f70-8350-b7865456ae46-metrics-tls") pod "dns-default-dgmn6" (UID: "98250d35-1d7d-4f70-8350-b7865456ae46") : secret "dns-default-metrics-tls" not found Jul 31 18:44:31.416862 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:31.416820 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:31.484283 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:31.484217 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:32.616928 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.616869 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9kv6f" podStartSLOduration=10.980559947 podStartE2EDuration="14.616853397s" totalImagesPullingTime="3.63629345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:23.23475316 +0000 UTC m=+39.872742961" watchObservedRunningTime="2026-07-31 18:44:32.616853397 +0000 UTC m=+49.254843217" Jul 31 18:44:32.617313 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.617163 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m76n7"] Jul 31 18:44:32.662818 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.662781 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m76n7"] Jul 31 18:44:32.662971 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.662893 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m76n7" Jul 31 18:44:32.667782 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.667759 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 31 18:44:32.667996 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.667971 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 31 18:44:32.668179 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.668162 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 31 18:44:32.668284 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.668195 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8rmqt\"" Jul 31 18:44:32.669766 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.669748 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 31 18:44:32.802696 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.802663 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-77696685b-rkrsg"] Jul 31 18:44:32.820658 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.820633 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/45c0e651-aba9-4741-84db-8a06cd82c46e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-m76n7\" (UID: \"45c0e651-aba9-4741-84db-8a06cd82c46e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m76n7" Jul 31 18:44:32.820797 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.820663 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv82j\" (UniqueName: \"kubernetes.io/projected/45c0e651-aba9-4741-84db-8a06cd82c46e-kube-api-access-rv82j\") pod \"cluster-monitoring-operator-6f55c4f58d-m76n7\" (UID: \"45c0e651-aba9-4741-84db-8a06cd82c46e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m76n7" Jul 31 18:44:32.820797 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.820697 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/45c0e651-aba9-4741-84db-8a06cd82c46e-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-m76n7\" (UID: \"45c0e651-aba9-4741-84db-8a06cd82c46e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m76n7" Jul 31 18:44:32.828288 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.828261 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-kbx2l"] Jul 31 18:44:32.828422 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.828333 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-rkrsg" Jul 31 18:44:32.833403 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.833383 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 31 18:44:32.834636 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.834615 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-k8vsq\"" Jul 31 18:44:32.834790 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.834769 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 31 18:44:32.834884 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.834773 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 31 18:44:32.845987 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.845966 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 31 18:44:32.846322 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.846293 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-rkrsg"] Jul 31 18:44:32.846395 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.846329 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-kbx2l"] Jul 31 18:44:32.846436 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.846416 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-kbx2l" Jul 31 18:44:32.848939 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.848923 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 31 18:44:32.849183 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.849157 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 31 18:44:32.849482 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.849466 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-pxd4x\"" Jul 31 18:44:32.856440 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.856424 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 31 18:44:32.863364 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.863350 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 31 18:44:32.901010 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.900947 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6bf57fb7b8-l6h5t"] Jul 31 18:44:32.921334 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.921301 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/45c0e651-aba9-4741-84db-8a06cd82c46e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-m76n7\" (UID: \"45c0e651-aba9-4741-84db-8a06cd82c46e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m76n7" Jul 31 18:44:32.921450 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.921336 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv82j\" (UniqueName: \"kubernetes.io/projected/45c0e651-aba9-4741-84db-8a06cd82c46e-kube-api-access-rv82j\") pod \"cluster-monitoring-operator-6f55c4f58d-m76n7\" (UID: \"45c0e651-aba9-4741-84db-8a06cd82c46e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m76n7" Jul 31 18:44:32.921450 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.921375 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24f12167-7a95-45d7-bc34-9cafeec2201e-tmp\") pod \"insights-operator-77696685b-rkrsg\" (UID: \"24f12167-7a95-45d7-bc34-9cafeec2201e\") " pod="openshift-insights/insights-operator-77696685b-rkrsg" Jul 31 18:44:32.921450 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.921422 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/45c0e651-aba9-4741-84db-8a06cd82c46e-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-m76n7\" (UID: \"45c0e651-aba9-4741-84db-8a06cd82c46e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m76n7" Jul 31 18:44:32.921553 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:32.921454 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 31 18:44:32.921585 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:32.921568 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45c0e651-aba9-4741-84db-8a06cd82c46e-cluster-monitoring-operator-tls podName:45c0e651-aba9-4741-84db-8a06cd82c46e nodeName:}" failed. No retries permitted until 2026-07-31 18:44:33.42155096 +0000 UTC m=+50.059540752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/45c0e651-aba9-4741-84db-8a06cd82c46e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-m76n7" (UID: "45c0e651-aba9-4741-84db-8a06cd82c46e") : secret "cluster-monitoring-operator-tls" not found Jul 31 18:44:32.921698 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.921673 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24f12167-7a95-45d7-bc34-9cafeec2201e-service-ca-bundle\") pod \"insights-operator-77696685b-rkrsg\" (UID: \"24f12167-7a95-45d7-bc34-9cafeec2201e\") " pod="openshift-insights/insights-operator-77696685b-rkrsg" Jul 31 18:44:32.921860 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.921718 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24f12167-7a95-45d7-bc34-9cafeec2201e-serving-cert\") pod \"insights-operator-77696685b-rkrsg\" (UID: \"24f12167-7a95-45d7-bc34-9cafeec2201e\") " pod="openshift-insights/insights-operator-77696685b-rkrsg" Jul 31 18:44:32.921860 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.921794 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/24f12167-7a95-45d7-bc34-9cafeec2201e-snapshots\") pod \"insights-operator-77696685b-rkrsg\" (UID: \"24f12167-7a95-45d7-bc34-9cafeec2201e\") " pod="openshift-insights/insights-operator-77696685b-rkrsg" Jul 31 18:44:32.921860 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.921850 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7h9k\" (UniqueName: \"kubernetes.io/projected/24f12167-7a95-45d7-bc34-9cafeec2201e-kube-api-access-v7h9k\") pod \"insights-operator-77696685b-rkrsg\" (UID: \"24f12167-7a95-45d7-bc34-9cafeec2201e\") " pod="openshift-insights/insights-operator-77696685b-rkrsg" Jul 31 18:44:32.921977 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.921888 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24f12167-7a95-45d7-bc34-9cafeec2201e-trusted-ca-bundle\") pod \"insights-operator-77696685b-rkrsg\" (UID: \"24f12167-7a95-45d7-bc34-9cafeec2201e\") " pod="openshift-insights/insights-operator-77696685b-rkrsg" Jul 31 18:44:32.922168 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.922150 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/45c0e651-aba9-4741-84db-8a06cd82c46e-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-m76n7\" (UID: \"45c0e651-aba9-4741-84db-8a06cd82c46e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m76n7" Jul 31 18:44:32.924436 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.924417 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-685dfffc96-vrmgp"] Jul 31 18:44:32.924551 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.924537 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" Jul 31 18:44:32.927452 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.927430 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 31 18:44:32.929339 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.927717 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 31 18:44:32.929339 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.928503 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 31 18:44:32.929665 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.929641 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4pbfx\"" Jul 31 18:44:32.944951 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.944921 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv82j\" (UniqueName: \"kubernetes.io/projected/45c0e651-aba9-4741-84db-8a06cd82c46e-kube-api-access-rv82j\") pod \"cluster-monitoring-operator-6f55c4f58d-m76n7\" (UID: \"45c0e651-aba9-4741-84db-8a06cd82c46e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m76n7" Jul 31 18:44:32.946675 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.946655 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 31 18:44:32.952538 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.952521 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-685dfffc96-vrmgp"] Jul 31 18:44:32.952612 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.952548 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bf57fb7b8-l6h5t"] Jul 31 18:44:32.952654 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.952640 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:44:32.955470 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.955450 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 31 18:44:32.955878 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.955861 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 31 18:44:32.955959 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.955894 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 31 18:44:32.956065 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.956049 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 31 18:44:32.956284 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.956271 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 31 18:44:32.956403 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.956388 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 31 18:44:32.957104 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:32.957089 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-s6rjx\"" Jul 31 18:44:33.023206 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.023173 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24f12167-7a95-45d7-bc34-9cafeec2201e-tmp\") pod \"insights-operator-77696685b-rkrsg\" (UID: \"24f12167-7a95-45d7-bc34-9cafeec2201e\") " pod="openshift-insights/insights-operator-77696685b-rkrsg" Jul 31 18:44:33.023206 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.023210 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-image-registry-private-configuration\") pod \"image-registry-6bf57fb7b8-l6h5t\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" Jul 31 18:44:33.023421 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.023234 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-registry-tls\") pod \"image-registry-6bf57fb7b8-l6h5t\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" Jul 31 18:44:33.023421 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.023302 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-registry-certificates\") pod \"image-registry-6bf57fb7b8-l6h5t\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" Jul 31 18:44:33.023421 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.023354 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24f12167-7a95-45d7-bc34-9cafeec2201e-service-ca-bundle\") pod \"insights-operator-77696685b-rkrsg\" (UID: \"24f12167-7a95-45d7-bc34-9cafeec2201e\") " pod="openshift-insights/insights-operator-77696685b-rkrsg" Jul 31 18:44:33.023421 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.023386 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24f12167-7a95-45d7-bc34-9cafeec2201e-serving-cert\") pod \"insights-operator-77696685b-rkrsg\" (UID: \"24f12167-7a95-45d7-bc34-9cafeec2201e\") " pod="openshift-insights/insights-operator-77696685b-rkrsg" Jul 31 18:44:33.023421 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.023409 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-installation-pull-secrets\") pod \"image-registry-6bf57fb7b8-l6h5t\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" Jul 31 18:44:33.023611 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.023441 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k49jz\" (UniqueName: \"kubernetes.io/projected/7c5dadcb-6709-4bad-9a34-d4bdfe44fce1-kube-api-access-k49jz\") pod \"cluster-samples-operator-559cbf5745-kbx2l\" (UID: \"7c5dadcb-6709-4bad-9a34-d4bdfe44fce1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-kbx2l" Jul 31 18:44:33.023611 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.023504 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-bound-sa-token\") pod \"image-registry-6bf57fb7b8-l6h5t\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" Jul 31 18:44:33.023611 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.023535 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/24f12167-7a95-45d7-bc34-9cafeec2201e-snapshots\") pod \"insights-operator-77696685b-rkrsg\" (UID: \"24f12167-7a95-45d7-bc34-9cafeec2201e\") " pod="openshift-insights/insights-operator-77696685b-rkrsg" Jul 31 18:44:33.023611 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.023559 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-trusted-ca\") pod \"image-registry-6bf57fb7b8-l6h5t\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" Jul 31 18:44:33.023611 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.023599 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-ca-trust-extracted\") pod \"image-registry-6bf57fb7b8-l6h5t\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" Jul 31 18:44:33.023791 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.023598 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24f12167-7a95-45d7-bc34-9cafeec2201e-tmp\") pod \"insights-operator-77696685b-rkrsg\" (UID: \"24f12167-7a95-45d7-bc34-9cafeec2201e\") " pod="openshift-insights/insights-operator-77696685b-rkrsg" Jul 31 18:44:33.023791 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.023640 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c5dadcb-6709-4bad-9a34-d4bdfe44fce1-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-kbx2l\" (UID: \"7c5dadcb-6709-4bad-9a34-d4bdfe44fce1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-kbx2l" Jul 31 18:44:33.023791 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.023678 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7h9k\" (UniqueName: \"kubernetes.io/projected/24f12167-7a95-45d7-bc34-9cafeec2201e-kube-api-access-v7h9k\") pod \"insights-operator-77696685b-rkrsg\" (UID: \"24f12167-7a95-45d7-bc34-9cafeec2201e\") " pod="openshift-insights/insights-operator-77696685b-rkrsg" Jul 31 18:44:33.023791 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.023700 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnsm5\" (UniqueName: \"kubernetes.io/projected/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-kube-api-access-jnsm5\") pod \"image-registry-6bf57fb7b8-l6h5t\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" Jul 31 18:44:33.023791 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.023730 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24f12167-7a95-45d7-bc34-9cafeec2201e-trusted-ca-bundle\") pod \"insights-operator-77696685b-rkrsg\" (UID: \"24f12167-7a95-45d7-bc34-9cafeec2201e\") " pod="openshift-insights/insights-operator-77696685b-rkrsg" Jul 31 18:44:33.024022 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.023923 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24f12167-7a95-45d7-bc34-9cafeec2201e-service-ca-bundle\") pod \"insights-operator-77696685b-rkrsg\" (UID: \"24f12167-7a95-45d7-bc34-9cafeec2201e\") " pod="openshift-insights/insights-operator-77696685b-rkrsg" Jul 31 18:44:33.024204 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.024178 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/24f12167-7a95-45d7-bc34-9cafeec2201e-snapshots\") pod \"insights-operator-77696685b-rkrsg\" (UID: \"24f12167-7a95-45d7-bc34-9cafeec2201e\") " pod="openshift-insights/insights-operator-77696685b-rkrsg" Jul 31 18:44:33.024728 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.024707 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24f12167-7a95-45d7-bc34-9cafeec2201e-trusted-ca-bundle\") pod \"insights-operator-77696685b-rkrsg\" (UID: \"24f12167-7a95-45d7-bc34-9cafeec2201e\") " pod="openshift-insights/insights-operator-77696685b-rkrsg" Jul 31 18:44:33.025775 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.025755 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24f12167-7a95-45d7-bc34-9cafeec2201e-serving-cert\") pod \"insights-operator-77696685b-rkrsg\" (UID: \"24f12167-7a95-45d7-bc34-9cafeec2201e\") " pod="openshift-insights/insights-operator-77696685b-rkrsg" Jul 31 18:44:33.034592 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.034573 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7h9k\" (UniqueName: \"kubernetes.io/projected/24f12167-7a95-45d7-bc34-9cafeec2201e-kube-api-access-v7h9k\") pod \"insights-operator-77696685b-rkrsg\" (UID: \"24f12167-7a95-45d7-bc34-9cafeec2201e\") " pod="openshift-insights/insights-operator-77696685b-rkrsg" Jul 31 18:44:33.124913 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.124891 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-installation-pull-secrets\") pod \"image-registry-6bf57fb7b8-l6h5t\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" Jul 31 18:44:33.125007 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.124924 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gp6m\" (UniqueName: \"kubernetes.io/projected/a8c11319-084e-4318-9e92-5a4a68d8c916-kube-api-access-2gp6m\") pod \"router-default-685dfffc96-vrmgp\" (UID: \"a8c11319-084e-4318-9e92-5a4a68d8c916\") " pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:44:33.125007 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.124943 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k49jz\" (UniqueName: \"kubernetes.io/projected/7c5dadcb-6709-4bad-9a34-d4bdfe44fce1-kube-api-access-k49jz\") pod \"cluster-samples-operator-559cbf5745-kbx2l\" (UID: \"7c5dadcb-6709-4bad-9a34-d4bdfe44fce1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-kbx2l" Jul 31 18:44:33.125097 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.125068 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnsm5\" (UniqueName: \"kubernetes.io/projected/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-kube-api-access-jnsm5\") pod \"image-registry-6bf57fb7b8-l6h5t\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" Jul 31 18:44:33.125141 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.125101 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-trusted-ca\") pod \"image-registry-6bf57fb7b8-l6h5t\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" Jul 31 18:44:33.125188 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.125138 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-ca-trust-extracted\") pod \"image-registry-6bf57fb7b8-l6h5t\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" Jul 31 18:44:33.125188 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.125166 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8c11319-084e-4318-9e92-5a4a68d8c916-service-ca-bundle\") pod \"router-default-685dfffc96-vrmgp\" (UID: \"a8c11319-084e-4318-9e92-5a4a68d8c916\") " pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:44:33.125318 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.125201 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-registry-tls\") pod \"image-registry-6bf57fb7b8-l6h5t\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" Jul 31 18:44:33.125318 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.125287 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a8c11319-084e-4318-9e92-5a4a68d8c916-default-certificate\") pod \"router-default-685dfffc96-vrmgp\" (UID: \"a8c11319-084e-4318-9e92-5a4a68d8c916\") " pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:44:33.125428 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.125343 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-registry-certificates\") pod \"image-registry-6bf57fb7b8-l6h5t\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" Jul 31 18:44:33.125428 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.125395 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a8c11319-084e-4318-9e92-5a4a68d8c916-stats-auth\") pod \"router-default-685dfffc96-vrmgp\" (UID: \"a8c11319-084e-4318-9e92-5a4a68d8c916\") " pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:44:33.125527 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.125453 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-bound-sa-token\") pod \"image-registry-6bf57fb7b8-l6h5t\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" Jul 31 18:44:33.125527 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.125481 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c5dadcb-6709-4bad-9a34-d4bdfe44fce1-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-kbx2l\" (UID: \"7c5dadcb-6709-4bad-9a34-d4bdfe44fce1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-kbx2l" Jul 31 18:44:33.125623 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.125545 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8c11319-084e-4318-9e92-5a4a68d8c916-metrics-certs\") pod \"router-default-685dfffc96-vrmgp\" (UID: \"a8c11319-084e-4318-9e92-5a4a68d8c916\") " pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:44:33.125623 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.125554 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-ca-trust-extracted\") pod \"image-registry-6bf57fb7b8-l6h5t\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" Jul 31 18:44:33.125623 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:33.125581 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 31 18:44:33.125765 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:33.125645 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c5dadcb-6709-4bad-9a34-d4bdfe44fce1-samples-operator-tls podName:7c5dadcb-6709-4bad-9a34-d4bdfe44fce1 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:33.625628292 +0000 UTC m=+50.263618076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7c5dadcb-6709-4bad-9a34-d4bdfe44fce1-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-kbx2l" (UID: "7c5dadcb-6709-4bad-9a34-d4bdfe44fce1") : secret "samples-operator-tls" not found Jul 31 18:44:33.125765 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.125636 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-image-registry-private-configuration\") pod \"image-registry-6bf57fb7b8-l6h5t\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" Jul 31 18:44:33.125765 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:33.125705 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 18:44:33.125765 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:33.125748 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bf57fb7b8-l6h5t: secret "image-registry-tls" not found Jul 31 18:44:33.125971 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:33.125804 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-registry-tls podName:cdc4eaf7-429f-4fc0-be49-1c4913cfdb85 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:33.62578844 +0000 UTC m=+50.263778227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-registry-tls") pod "image-registry-6bf57fb7b8-l6h5t" (UID: "cdc4eaf7-429f-4fc0-be49-1c4913cfdb85") : secret "image-registry-tls" not found Jul 31 18:44:33.126042 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.125996 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-registry-certificates\") pod \"image-registry-6bf57fb7b8-l6h5t\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" Jul 31 18:44:33.126324 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.126295 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:33.126934 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.126914 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-trusted-ca\") pod \"image-registry-6bf57fb7b8-l6h5t\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" Jul 31 18:44:33.127515 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.127490 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-installation-pull-secrets\") pod \"image-registry-6bf57fb7b8-l6h5t\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" Jul 31 18:44:33.128290 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.128272 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-image-registry-private-configuration\") pod \"image-registry-6bf57fb7b8-l6h5t\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" Jul 31 18:44:33.134692 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.134669 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k49jz\" (UniqueName: \"kubernetes.io/projected/7c5dadcb-6709-4bad-9a34-d4bdfe44fce1-kube-api-access-k49jz\") pod \"cluster-samples-operator-559cbf5745-kbx2l\" (UID: \"7c5dadcb-6709-4bad-9a34-d4bdfe44fce1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-kbx2l" Jul 31 18:44:33.134764 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.134691 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnsm5\" (UniqueName: \"kubernetes.io/projected/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-kube-api-access-jnsm5\") pod \"image-registry-6bf57fb7b8-l6h5t\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" Jul 31 18:44:33.134825 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.134809 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-bound-sa-token\") pod \"image-registry-6bf57fb7b8-l6h5t\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" Jul 31 18:44:33.136714 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.136701 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-rkrsg" Jul 31 18:44:33.226415 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.226269 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8c11319-084e-4318-9e92-5a4a68d8c916-service-ca-bundle\") pod \"router-default-685dfffc96-vrmgp\" (UID: \"a8c11319-084e-4318-9e92-5a4a68d8c916\") " pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:44:33.226415 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.226340 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a8c11319-084e-4318-9e92-5a4a68d8c916-default-certificate\") pod \"router-default-685dfffc96-vrmgp\" (UID: \"a8c11319-084e-4318-9e92-5a4a68d8c916\") " pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:44:33.226415 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.226390 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a8c11319-084e-4318-9e92-5a4a68d8c916-stats-auth\") pod \"router-default-685dfffc96-vrmgp\" (UID: \"a8c11319-084e-4318-9e92-5a4a68d8c916\") " pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:44:33.226660 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.226446 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:33.226660 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:33.226467 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a8c11319-084e-4318-9e92-5a4a68d8c916-service-ca-bundle podName:a8c11319-084e-4318-9e92-5a4a68d8c916 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:33.726443086 +0000 UTC m=+50.364432884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a8c11319-084e-4318-9e92-5a4a68d8c916-service-ca-bundle") pod "router-default-685dfffc96-vrmgp" (UID: "a8c11319-084e-4318-9e92-5a4a68d8c916") : configmap references non-existent config key: service-ca.crt Jul 31 18:44:33.226660 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.226497 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8c11319-084e-4318-9e92-5a4a68d8c916-metrics-certs\") pod \"router-default-685dfffc96-vrmgp\" (UID: \"a8c11319-084e-4318-9e92-5a4a68d8c916\") " pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:44:33.226660 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.226568 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gp6m\" (UniqueName: \"kubernetes.io/projected/a8c11319-084e-4318-9e92-5a4a68d8c916-kube-api-access-2gp6m\") pod \"router-default-685dfffc96-vrmgp\" (UID: \"a8c11319-084e-4318-9e92-5a4a68d8c916\") " pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:44:33.227037 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:33.226945 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 31 18:44:33.227037 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:33.227003 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8c11319-084e-4318-9e92-5a4a68d8c916-metrics-certs podName:a8c11319-084e-4318-9e92-5a4a68d8c916 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:33.726986315 +0000 UTC m=+50.364976106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8c11319-084e-4318-9e92-5a4a68d8c916-metrics-certs") pod "router-default-685dfffc96-vrmgp" (UID: "a8c11319-084e-4318-9e92-5a4a68d8c916") : secret "router-metrics-certs-default" not found Jul 31 18:44:33.229479 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.229434 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a8c11319-084e-4318-9e92-5a4a68d8c916-stats-auth\") pod \"router-default-685dfffc96-vrmgp\" (UID: \"a8c11319-084e-4318-9e92-5a4a68d8c916\") " pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:44:33.229851 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.229832 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a8c11319-084e-4318-9e92-5a4a68d8c916-default-certificate\") pod \"router-default-685dfffc96-vrmgp\" (UID: \"a8c11319-084e-4318-9e92-5a4a68d8c916\") " pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:44:33.235335 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.235311 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gp6m\" (UniqueName: \"kubernetes.io/projected/a8c11319-084e-4318-9e92-5a4a68d8c916-kube-api-access-2gp6m\") pod \"router-default-685dfffc96-vrmgp\" (UID: \"a8c11319-084e-4318-9e92-5a4a68d8c916\") " pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:44:33.294711 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.294682 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-rkrsg"] Jul 31 18:44:33.297742 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:44:33.297715 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f12167_7a95_45d7_bc34_9cafeec2201e.slice/crio-440cf578f9f0842d0f7aa0abbc5343d25c4e42b87c87b6a48161eda052d0d0ba WatchSource:0}: Error finding container 440cf578f9f0842d0f7aa0abbc5343d25c4e42b87c87b6a48161eda052d0d0ba: Status 404 returned error can't find the container with id 440cf578f9f0842d0f7aa0abbc5343d25c4e42b87c87b6a48161eda052d0d0ba Jul 31 18:44:33.429343 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.429268 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/45c0e651-aba9-4741-84db-8a06cd82c46e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-m76n7\" (UID: \"45c0e651-aba9-4741-84db-8a06cd82c46e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m76n7" Jul 31 18:44:33.429473 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:33.429432 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 31 18:44:33.429511 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:33.429492 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45c0e651-aba9-4741-84db-8a06cd82c46e-cluster-monitoring-operator-tls podName:45c0e651-aba9-4741-84db-8a06cd82c46e nodeName:}" failed. No retries permitted until 2026-07-31 18:44:34.429478754 +0000 UTC m=+51.067468531 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/45c0e651-aba9-4741-84db-8a06cd82c46e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-m76n7" (UID: "45c0e651-aba9-4741-84db-8a06cd82c46e") : secret "cluster-monitoring-operator-tls" not found Jul 31 18:44:33.610840 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.610810 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-6txtg"] Jul 31 18:44:33.630801 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.630777 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-registry-tls\") pod \"image-registry-6bf57fb7b8-l6h5t\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" Jul 31 18:44:33.631131 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.630847 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c5dadcb-6709-4bad-9a34-d4bdfe44fce1-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-kbx2l\" (UID: \"7c5dadcb-6709-4bad-9a34-d4bdfe44fce1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-kbx2l" Jul 31 18:44:33.631131 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:33.630914 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 18:44:33.631131 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:33.630932 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 31 18:44:33.631131 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:33.630940 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bf57fb7b8-l6h5t: secret "image-registry-tls" not found Jul 31 18:44:33.631131 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:33.630985 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c5dadcb-6709-4bad-9a34-d4bdfe44fce1-samples-operator-tls podName:7c5dadcb-6709-4bad-9a34-d4bdfe44fce1 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:34.630967247 +0000 UTC m=+51.268957025 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7c5dadcb-6709-4bad-9a34-d4bdfe44fce1-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-kbx2l" (UID: "7c5dadcb-6709-4bad-9a34-d4bdfe44fce1") : secret "samples-operator-tls" not found Jul 31 18:44:33.631131 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:33.631002 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-registry-tls podName:cdc4eaf7-429f-4fc0-be49-1c4913cfdb85 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:34.630993732 +0000 UTC m=+51.268983509 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-registry-tls") pod "image-registry-6bf57fb7b8-l6h5t" (UID: "cdc4eaf7-429f-4fc0-be49-1c4913cfdb85") : secret "image-registry-tls" not found Jul 31 18:44:33.641856 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.641840 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-6txtg"] Jul 31 18:44:33.641931 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.641916 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6txtg" Jul 31 18:44:33.643767 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.643743 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 31 18:44:33.643896 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.643766 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 31 18:44:33.643896 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.643814 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 31 18:44:33.643896 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.643832 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-44tg4\"" Jul 31 18:44:33.644036 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.643974 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 31 18:44:33.731644 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.731581 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8c11319-084e-4318-9e92-5a4a68d8c916-metrics-certs\") pod \"router-default-685dfffc96-vrmgp\" (UID: \"a8c11319-084e-4318-9e92-5a4a68d8c916\") " pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:44:33.731644 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.731641 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8c11319-084e-4318-9e92-5a4a68d8c916-service-ca-bundle\") pod \"router-default-685dfffc96-vrmgp\" (UID: \"a8c11319-084e-4318-9e92-5a4a68d8c916\") " pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:44:33.731801 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:33.731728 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 31 18:44:33.731801 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:33.731778 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a8c11319-084e-4318-9e92-5a4a68d8c916-service-ca-bundle podName:a8c11319-084e-4318-9e92-5a4a68d8c916 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:34.731766129 +0000 UTC m=+51.369755907 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a8c11319-084e-4318-9e92-5a4a68d8c916-service-ca-bundle") pod "router-default-685dfffc96-vrmgp" (UID: "a8c11319-084e-4318-9e92-5a4a68d8c916") : configmap references non-existent config key: service-ca.crt Jul 31 18:44:33.731801 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:33.731793 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8c11319-084e-4318-9e92-5a4a68d8c916-metrics-certs podName:a8c11319-084e-4318-9e92-5a4a68d8c916 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:34.731787464 +0000 UTC m=+51.369777241 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8c11319-084e-4318-9e92-5a4a68d8c916-metrics-certs") pod "router-default-685dfffc96-vrmgp" (UID: "a8c11319-084e-4318-9e92-5a4a68d8c916") : secret "router-metrics-certs-default" not found Jul 31 18:44:33.832794 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.832764 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80e51b4d-e2a8-4f98-a568-1c21551ed35c-serving-cert\") pod \"service-ca-operator-784f97bcf6-6txtg\" (UID: \"80e51b4d-e2a8-4f98-a568-1c21551ed35c\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6txtg" Jul 31 18:44:33.832794 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.832794 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pkft\" (UniqueName: \"kubernetes.io/projected/80e51b4d-e2a8-4f98-a568-1c21551ed35c-kube-api-access-2pkft\") pod \"service-ca-operator-784f97bcf6-6txtg\" (UID: \"80e51b4d-e2a8-4f98-a568-1c21551ed35c\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6txtg" Jul 31 18:44:33.832969 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.832921 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80e51b4d-e2a8-4f98-a568-1c21551ed35c-config\") pod \"service-ca-operator-784f97bcf6-6txtg\" (UID: \"80e51b4d-e2a8-4f98-a568-1c21551ed35c\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6txtg" Jul 31 18:44:33.933710 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.933674 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80e51b4d-e2a8-4f98-a568-1c21551ed35c-serving-cert\") pod \"service-ca-operator-784f97bcf6-6txtg\" (UID: \"80e51b4d-e2a8-4f98-a568-1c21551ed35c\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6txtg" Jul 31 18:44:33.933899 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.933716 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pkft\" (UniqueName: \"kubernetes.io/projected/80e51b4d-e2a8-4f98-a568-1c21551ed35c-kube-api-access-2pkft\") pod \"service-ca-operator-784f97bcf6-6txtg\" (UID: \"80e51b4d-e2a8-4f98-a568-1c21551ed35c\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6txtg" Jul 31 18:44:33.933899 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.933847 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80e51b4d-e2a8-4f98-a568-1c21551ed35c-config\") pod \"service-ca-operator-784f97bcf6-6txtg\" (UID: \"80e51b4d-e2a8-4f98-a568-1c21551ed35c\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6txtg" Jul 31 18:44:33.934383 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.934361 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80e51b4d-e2a8-4f98-a568-1c21551ed35c-config\") pod \"service-ca-operator-784f97bcf6-6txtg\" (UID: \"80e51b4d-e2a8-4f98-a568-1c21551ed35c\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6txtg" Jul 31 18:44:33.936124 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.936095 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80e51b4d-e2a8-4f98-a568-1c21551ed35c-serving-cert\") pod \"service-ca-operator-784f97bcf6-6txtg\" (UID: \"80e51b4d-e2a8-4f98-a568-1c21551ed35c\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6txtg" Jul 31 18:44:33.942481 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.942457 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pkft\" (UniqueName: \"kubernetes.io/projected/80e51b4d-e2a8-4f98-a568-1c21551ed35c-kube-api-access-2pkft\") pod \"service-ca-operator-784f97bcf6-6txtg\" (UID: \"80e51b4d-e2a8-4f98-a568-1c21551ed35c\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6txtg" Jul 31 18:44:33.950331 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:33.950306 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6txtg" Jul 31 18:44:34.118907 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:34.118873 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-6txtg"] Jul 31 18:44:34.122842 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:44:34.122813 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e51b4d_e2a8_4f98_a568_1c21551ed35c.slice/crio-720b182d60218f556e92b17a7d30e97d8f2fef3a3d69ab641c84eae41fa4d850 WatchSource:0}: Error finding container 720b182d60218f556e92b17a7d30e97d8f2fef3a3d69ab641c84eae41fa4d850: Status 404 returned error can't find the container with id 720b182d60218f556e92b17a7d30e97d8f2fef3a3d69ab641c84eae41fa4d850 Jul 31 18:44:34.192313 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:34.192264 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6txtg" event={"ID":"80e51b4d-e2a8-4f98-a568-1c21551ed35c","Type":"ContainerStarted","Data":"720b182d60218f556e92b17a7d30e97d8f2fef3a3d69ab641c84eae41fa4d850"} Jul 31 18:44:34.193522 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:34.193492 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-rkrsg" event={"ID":"24f12167-7a95-45d7-bc34-9cafeec2201e","Type":"ContainerStarted","Data":"440cf578f9f0842d0f7aa0abbc5343d25c4e42b87c87b6a48161eda052d0d0ba"} Jul 31 18:44:34.359401 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:34.359322 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:34.438523 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:34.438490 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/45c0e651-aba9-4741-84db-8a06cd82c46e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-m76n7\" (UID: \"45c0e651-aba9-4741-84db-8a06cd82c46e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m76n7" Jul 31 18:44:34.438665 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:34.438644 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 31 18:44:34.438724 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:34.438712 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45c0e651-aba9-4741-84db-8a06cd82c46e-cluster-monitoring-operator-tls podName:45c0e651-aba9-4741-84db-8a06cd82c46e nodeName:}" failed. No retries permitted until 2026-07-31 18:44:36.438698065 +0000 UTC m=+53.076687843 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/45c0e651-aba9-4741-84db-8a06cd82c46e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-m76n7" (UID: "45c0e651-aba9-4741-84db-8a06cd82c46e") : secret "cluster-monitoring-operator-tls" not found Jul 31 18:44:34.454167 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:34.454136 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:34.533257 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:34.533205 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:34.640208 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:34.640124 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-registry-tls\") pod \"image-registry-6bf57fb7b8-l6h5t\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" Jul 31 18:44:34.640661 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:34.640216 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c5dadcb-6709-4bad-9a34-d4bdfe44fce1-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-kbx2l\" (UID: \"7c5dadcb-6709-4bad-9a34-d4bdfe44fce1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-kbx2l" Jul 31 18:44:34.640661 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:34.640271 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 18:44:34.640661 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:34.640294 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bf57fb7b8-l6h5t: secret "image-registry-tls" not found Jul 31 18:44:34.640661 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:34.640354 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-registry-tls podName:cdc4eaf7-429f-4fc0-be49-1c4913cfdb85 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:36.640336051 +0000 UTC m=+53.278325831 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-registry-tls") pod "image-registry-6bf57fb7b8-l6h5t" (UID: "cdc4eaf7-429f-4fc0-be49-1c4913cfdb85") : secret "image-registry-tls" not found Jul 31 18:44:34.640661 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:34.640405 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 31 18:44:34.640661 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:34.640467 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c5dadcb-6709-4bad-9a34-d4bdfe44fce1-samples-operator-tls podName:7c5dadcb-6709-4bad-9a34-d4bdfe44fce1 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:36.640450972 +0000 UTC m=+53.278440751 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7c5dadcb-6709-4bad-9a34-d4bdfe44fce1-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-kbx2l" (UID: "7c5dadcb-6709-4bad-9a34-d4bdfe44fce1") : secret "samples-operator-tls" not found Jul 31 18:44:34.741143 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:34.741106 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8c11319-084e-4318-9e92-5a4a68d8c916-service-ca-bundle\") pod \"router-default-685dfffc96-vrmgp\" (UID: \"a8c11319-084e-4318-9e92-5a4a68d8c916\") " pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:44:34.741340 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:34.741192 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98250d35-1d7d-4f70-8350-b7865456ae46-metrics-tls\") pod \"dns-default-dgmn6\" (UID: \"98250d35-1d7d-4f70-8350-b7865456ae46\") " pod="openshift-dns/dns-default-dgmn6" Jul 31 18:44:34.741340 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:34.741282 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8c11319-084e-4318-9e92-5a4a68d8c916-metrics-certs\") pod \"router-default-685dfffc96-vrmgp\" (UID: \"a8c11319-084e-4318-9e92-5a4a68d8c916\") " pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:44:34.741454 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:34.741379 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a8c11319-084e-4318-9e92-5a4a68d8c916-service-ca-bundle podName:a8c11319-084e-4318-9e92-5a4a68d8c916 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:36.741356248 +0000 UTC m=+53.379346047 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a8c11319-084e-4318-9e92-5a4a68d8c916-service-ca-bundle") pod "router-default-685dfffc96-vrmgp" (UID: "a8c11319-084e-4318-9e92-5a4a68d8c916") : configmap references non-existent config key: service-ca.crt Jul 31 18:44:34.741518 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:34.741456 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 31 18:44:34.741518 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:34.741509 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8c11319-084e-4318-9e92-5a4a68d8c916-metrics-certs podName:a8c11319-084e-4318-9e92-5a4a68d8c916 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:36.741493568 +0000 UTC m=+53.379483349 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8c11319-084e-4318-9e92-5a4a68d8c916-metrics-certs") pod "router-default-685dfffc96-vrmgp" (UID: "a8c11319-084e-4318-9e92-5a4a68d8c916") : secret "router-metrics-certs-default" not found Jul 31 18:44:34.741612 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:34.741570 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 18:44:34.741685 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:34.741666 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98250d35-1d7d-4f70-8350-b7865456ae46-metrics-tls podName:98250d35-1d7d-4f70-8350-b7865456ae46 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:50.741654495 +0000 UTC m=+67.379644276 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98250d35-1d7d-4f70-8350-b7865456ae46-metrics-tls") pod "dns-default-dgmn6" (UID: "98250d35-1d7d-4f70-8350-b7865456ae46") : secret "dns-default-metrics-tls" not found Jul 31 18:44:35.337940 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:35.337895 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:35.437821 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:35.437783 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:36.288115 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:36.288075 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:36.454846 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:36.454809 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/45c0e651-aba9-4741-84db-8a06cd82c46e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-m76n7\" (UID: \"45c0e651-aba9-4741-84db-8a06cd82c46e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m76n7" Jul 31 18:44:36.455048 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:36.454945 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 31 18:44:36.455048 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:36.455026 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45c0e651-aba9-4741-84db-8a06cd82c46e-cluster-monitoring-operator-tls podName:45c0e651-aba9-4741-84db-8a06cd82c46e nodeName:}" failed. No retries permitted until 2026-07-31 18:44:40.455006448 +0000 UTC m=+57.092996231 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/45c0e651-aba9-4741-84db-8a06cd82c46e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-m76n7" (UID: "45c0e651-aba9-4741-84db-8a06cd82c46e") : secret "cluster-monitoring-operator-tls" not found Jul 31 18:44:36.656341 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:36.656271 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-registry-tls\") pod \"image-registry-6bf57fb7b8-l6h5t\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" Jul 31 18:44:36.656341 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:36.656341 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c5dadcb-6709-4bad-9a34-d4bdfe44fce1-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-kbx2l\" (UID: \"7c5dadcb-6709-4bad-9a34-d4bdfe44fce1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-kbx2l" Jul 31 18:44:36.656505 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:36.656438 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 18:44:36.656505 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:36.656464 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bf57fb7b8-l6h5t: secret "image-registry-tls" not found Jul 31 18:44:36.656505 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:36.656444 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 31 18:44:36.656605 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:36.656517 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c5dadcb-6709-4bad-9a34-d4bdfe44fce1-samples-operator-tls podName:7c5dadcb-6709-4bad-9a34-d4bdfe44fce1 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:40.656502006 +0000 UTC m=+57.294491784 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7c5dadcb-6709-4bad-9a34-d4bdfe44fce1-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-kbx2l" (UID: "7c5dadcb-6709-4bad-9a34-d4bdfe44fce1") : secret "samples-operator-tls" not found Jul 31 18:44:36.656605 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:36.656530 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-registry-tls podName:cdc4eaf7-429f-4fc0-be49-1c4913cfdb85 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:40.656524626 +0000 UTC m=+57.294514404 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-registry-tls") pod "image-registry-6bf57fb7b8-l6h5t" (UID: "cdc4eaf7-429f-4fc0-be49-1c4913cfdb85") : secret "image-registry-tls" not found Jul 31 18:44:36.757390 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:36.757360 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8c11319-084e-4318-9e92-5a4a68d8c916-metrics-certs\") pod \"router-default-685dfffc96-vrmgp\" (UID: \"a8c11319-084e-4318-9e92-5a4a68d8c916\") " pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:44:36.757493 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:36.757474 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8c11319-084e-4318-9e92-5a4a68d8c916-service-ca-bundle\") pod \"router-default-685dfffc96-vrmgp\" (UID: \"a8c11319-084e-4318-9e92-5a4a68d8c916\") " pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:44:36.757534 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:36.757499 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 31 18:44:36.757591 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:36.757554 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8c11319-084e-4318-9e92-5a4a68d8c916-metrics-certs podName:a8c11319-084e-4318-9e92-5a4a68d8c916 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:40.757540552 +0000 UTC m=+57.395530329 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8c11319-084e-4318-9e92-5a4a68d8c916-metrics-certs") pod "router-default-685dfffc96-vrmgp" (UID: "a8c11319-084e-4318-9e92-5a4a68d8c916") : secret "router-metrics-certs-default" not found Jul 31 18:44:36.757695 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:36.757677 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a8c11319-084e-4318-9e92-5a4a68d8c916-service-ca-bundle podName:a8c11319-084e-4318-9e92-5a4a68d8c916 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:40.757657495 +0000 UTC m=+57.395647296 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a8c11319-084e-4318-9e92-5a4a68d8c916-service-ca-bundle") pod "router-default-685dfffc96-vrmgp" (UID: "a8c11319-084e-4318-9e92-5a4a68d8c916") : configmap references non-existent config key: service-ca.crt Jul 31 18:44:37.202593 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:37.202559 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6txtg" event={"ID":"80e51b4d-e2a8-4f98-a568-1c21551ed35c","Type":"ContainerStarted","Data":"dfbd6125afe7dd3a77273523884317ec513d1753f95834df19bfbaca1046fae5"} Jul 31 18:44:37.204016 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:37.203990 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-rkrsg" event={"ID":"24f12167-7a95-45d7-bc34-9cafeec2201e","Type":"ContainerStarted","Data":"9f0aa2dfc2225c3b46f0b3fc2585f12b4c2eb61bd152a65939d07ae7cf140548"} Jul 31 18:44:37.227179 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:37.227136 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6txtg" podStartSLOduration=1.3119169 podStartE2EDuration="4.227124576s" totalImagesPullingTime="2.915207676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:44:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:37.225997412 +0000 UTC m=+53.863987211" watchObservedRunningTime="2026-07-31 18:44:37.227124576 +0000 UTC m=+53.865114411" Jul 31 18:44:37.255653 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:37.255604 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-77696685b-rkrsg" podStartSLOduration=1.861784879 podStartE2EDuration="5.255593798s" totalImagesPullingTime="3.393808919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:44:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:37.254657082 +0000 UTC m=+53.892646883" watchObservedRunningTime="2026-07-31 18:44:37.255593798 +0000 UTC m=+53.893583598" Jul 31 18:44:38.031205 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:38.031170 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vrdbp"] Jul 31 18:44:38.043232 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:38.043193 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:38.061192 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:38.061163 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vrdbp"] Jul 31 18:44:38.061322 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:38.061275 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vrdbp" Jul 31 18:44:38.063422 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:38.063399 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 31 18:44:38.063422 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:38.063407 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 31 18:44:38.063618 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:38.063566 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-v7tf4\"" Jul 31 18:44:38.071111 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:38.071091 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0e664160-fd29-4f67-b32e-8ae37b2641be-crio-socket\") pod \"insights-runtime-extractor-vrdbp\" (UID: \"0e664160-fd29-4f67-b32e-8ae37b2641be\") " pod="openshift-insights/insights-runtime-extractor-vrdbp" Jul 31 18:44:38.071223 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:38.071120 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnvvb\" (UniqueName: \"kubernetes.io/projected/0e664160-fd29-4f67-b32e-8ae37b2641be-kube-api-access-hnvvb\") pod \"insights-runtime-extractor-vrdbp\" (UID: \"0e664160-fd29-4f67-b32e-8ae37b2641be\") " pod="openshift-insights/insights-runtime-extractor-vrdbp" Jul 31 18:44:38.071223 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:38.071140 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e664160-fd29-4f67-b32e-8ae37b2641be-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vrdbp\" (UID: \"0e664160-fd29-4f67-b32e-8ae37b2641be\") " pod="openshift-insights/insights-runtime-extractor-vrdbp" Jul 31 18:44:38.071223 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:38.071188 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e664160-fd29-4f67-b32e-8ae37b2641be-data-volume\") pod \"insights-runtime-extractor-vrdbp\" (UID: \"0e664160-fd29-4f67-b32e-8ae37b2641be\") " pod="openshift-insights/insights-runtime-extractor-vrdbp" Jul 31 18:44:38.071409 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:38.071305 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e664160-fd29-4f67-b32e-8ae37b2641be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vrdbp\" (UID: \"0e664160-fd29-4f67-b32e-8ae37b2641be\") " pod="openshift-insights/insights-runtime-extractor-vrdbp" Jul 31 18:44:38.120675 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:38.120638 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:38.171691 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:38.171662 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e664160-fd29-4f67-b32e-8ae37b2641be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vrdbp\" (UID: \"0e664160-fd29-4f67-b32e-8ae37b2641be\") " pod="openshift-insights/insights-runtime-extractor-vrdbp" Jul 31 18:44:38.171823 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:38.171728 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0e664160-fd29-4f67-b32e-8ae37b2641be-crio-socket\") pod \"insights-runtime-extractor-vrdbp\" (UID: \"0e664160-fd29-4f67-b32e-8ae37b2641be\") " pod="openshift-insights/insights-runtime-extractor-vrdbp" Jul 31 18:44:38.171823 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:38.171750 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnvvb\" (UniqueName: \"kubernetes.io/projected/0e664160-fd29-4f67-b32e-8ae37b2641be-kube-api-access-hnvvb\") pod \"insights-runtime-extractor-vrdbp\" (UID: \"0e664160-fd29-4f67-b32e-8ae37b2641be\") " pod="openshift-insights/insights-runtime-extractor-vrdbp" Jul 31 18:44:38.171823 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:38.171767 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e664160-fd29-4f67-b32e-8ae37b2641be-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vrdbp\" (UID: \"0e664160-fd29-4f67-b32e-8ae37b2641be\") " pod="openshift-insights/insights-runtime-extractor-vrdbp" Jul 31 18:44:38.171823 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:38.171799 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e664160-fd29-4f67-b32e-8ae37b2641be-data-volume\") pod \"insights-runtime-extractor-vrdbp\" (UID: \"0e664160-fd29-4f67-b32e-8ae37b2641be\") " pod="openshift-insights/insights-runtime-extractor-vrdbp" Jul 31 18:44:38.171823 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:38.171804 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 18:44:38.172066 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:38.171865 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e664160-fd29-4f67-b32e-8ae37b2641be-insights-runtime-extractor-tls podName:0e664160-fd29-4f67-b32e-8ae37b2641be nodeName:}" failed. No retries permitted until 2026-07-31 18:44:38.671847148 +0000 UTC m=+55.309836926 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e664160-fd29-4f67-b32e-8ae37b2641be-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vrdbp" (UID: "0e664160-fd29-4f67-b32e-8ae37b2641be") : secret "insights-runtime-extractor-tls" not found Jul 31 18:44:38.172066 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:38.171880 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 18:44:38.172066 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:38.171937 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e664160-fd29-4f67-b32e-8ae37b2641be-kube-rbac-proxy-cm podName:0e664160-fd29-4f67-b32e-8ae37b2641be nodeName:}" failed. No retries permitted until 2026-07-31 18:44:38.67192035 +0000 UTC m=+55.309910128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0e664160-fd29-4f67-b32e-8ae37b2641be-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vrdbp" (UID: "0e664160-fd29-4f67-b32e-8ae37b2641be") : configmap "kube-rbac-proxy" not found Jul 31 18:44:38.172066 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:38.172015 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0e664160-fd29-4f67-b32e-8ae37b2641be-crio-socket\") pod \"insights-runtime-extractor-vrdbp\" (UID: \"0e664160-fd29-4f67-b32e-8ae37b2641be\") " pod="openshift-insights/insights-runtime-extractor-vrdbp" Jul 31 18:44:38.172301 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:38.172101 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e664160-fd29-4f67-b32e-8ae37b2641be-data-volume\") pod \"insights-runtime-extractor-vrdbp\" (UID: \"0e664160-fd29-4f67-b32e-8ae37b2641be\") " pod="openshift-insights/insights-runtime-extractor-vrdbp" Jul 31 18:44:38.187313 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:38.187288 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnvvb\" (UniqueName: \"kubernetes.io/projected/0e664160-fd29-4f67-b32e-8ae37b2641be-kube-api-access-hnvvb\") pod \"insights-runtime-extractor-vrdbp\" (UID: \"0e664160-fd29-4f67-b32e-8ae37b2641be\") " pod="openshift-insights/insights-runtime-extractor-vrdbp" Jul 31 18:44:38.582319 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:38.582283 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:38.641805 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:38.641778 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:38.676261 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:38.676219 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e664160-fd29-4f67-b32e-8ae37b2641be-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vrdbp\" (UID: \"0e664160-fd29-4f67-b32e-8ae37b2641be\") " pod="openshift-insights/insights-runtime-extractor-vrdbp" Jul 31 18:44:38.676367 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:38.676348 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 18:44:38.676446 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:38.676435 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e664160-fd29-4f67-b32e-8ae37b2641be-kube-rbac-proxy-cm podName:0e664160-fd29-4f67-b32e-8ae37b2641be nodeName:}" failed. No retries permitted until 2026-07-31 18:44:39.676415768 +0000 UTC m=+56.314405560 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0e664160-fd29-4f67-b32e-8ae37b2641be-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vrdbp" (UID: "0e664160-fd29-4f67-b32e-8ae37b2641be") : configmap "kube-rbac-proxy" not found Jul 31 18:44:38.676521 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:38.676489 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e664160-fd29-4f67-b32e-8ae37b2641be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vrdbp\" (UID: \"0e664160-fd29-4f67-b32e-8ae37b2641be\") " pod="openshift-insights/insights-runtime-extractor-vrdbp" Jul 31 18:44:38.676651 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:38.676633 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 18:44:38.676716 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:38.676705 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e664160-fd29-4f67-b32e-8ae37b2641be-insights-runtime-extractor-tls podName:0e664160-fd29-4f67-b32e-8ae37b2641be nodeName:}" failed. No retries permitted until 2026-07-31 18:44:39.676688527 +0000 UTC m=+56.314678310 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e664160-fd29-4f67-b32e-8ae37b2641be-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vrdbp" (UID: "0e664160-fd29-4f67-b32e-8ae37b2641be") : secret "insights-runtime-extractor-tls" not found Jul 31 18:44:39.165926 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:39.165892 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:39.232884 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:39.232853 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:39.551594 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:39.551566 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fjbvh_ba85de76-d297-4a7b-bb30-9c12572d9811/dns-node-resolver/0.log" Jul 31 18:44:39.596435 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:39.596401 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:39.685014 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:39.684981 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e664160-fd29-4f67-b32e-8ae37b2641be-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vrdbp\" (UID: \"0e664160-fd29-4f67-b32e-8ae37b2641be\") " pod="openshift-insights/insights-runtime-extractor-vrdbp" Jul 31 18:44:39.685146 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:39.685096 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 18:44:39.685146 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:39.685114 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e664160-fd29-4f67-b32e-8ae37b2641be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vrdbp\" (UID: \"0e664160-fd29-4f67-b32e-8ae37b2641be\") " pod="openshift-insights/insights-runtime-extractor-vrdbp" Jul 31 18:44:39.685281 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:39.685156 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e664160-fd29-4f67-b32e-8ae37b2641be-kube-rbac-proxy-cm podName:0e664160-fd29-4f67-b32e-8ae37b2641be nodeName:}" failed. No retries permitted until 2026-07-31 18:44:41.685143247 +0000 UTC m=+58.323133024 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0e664160-fd29-4f67-b32e-8ae37b2641be-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vrdbp" (UID: "0e664160-fd29-4f67-b32e-8ae37b2641be") : configmap "kube-rbac-proxy" not found Jul 31 18:44:39.685281 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:39.685222 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 18:44:39.685370 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:39.685294 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e664160-fd29-4f67-b32e-8ae37b2641be-insights-runtime-extractor-tls podName:0e664160-fd29-4f67-b32e-8ae37b2641be nodeName:}" failed. No retries permitted until 2026-07-31 18:44:41.685278012 +0000 UTC m=+58.323267803 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e664160-fd29-4f67-b32e-8ae37b2641be-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vrdbp" (UID: "0e664160-fd29-4f67-b32e-8ae37b2641be") : secret "insights-runtime-extractor-tls" not found Jul 31 18:44:39.722995 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:39.722968 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:39.809031 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:39.808943 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:39.906332 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:39.906301 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:40.491615 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:40.491581 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/45c0e651-aba9-4741-84db-8a06cd82c46e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-m76n7\" (UID: \"45c0e651-aba9-4741-84db-8a06cd82c46e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m76n7" Jul 31 18:44:40.491968 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:40.491720 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 31 18:44:40.491968 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:40.491778 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45c0e651-aba9-4741-84db-8a06cd82c46e-cluster-monitoring-operator-tls podName:45c0e651-aba9-4741-84db-8a06cd82c46e nodeName:}" failed. No retries permitted until 2026-07-31 18:44:48.491763294 +0000 UTC m=+65.129753075 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/45c0e651-aba9-4741-84db-8a06cd82c46e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-m76n7" (UID: "45c0e651-aba9-4741-84db-8a06cd82c46e") : secret "cluster-monitoring-operator-tls" not found Jul 31 18:44:40.550808 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:40.550782 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sbmpv_fa723eb5-7c9f-4676-aeda-50b07733ff96/node-ca/0.log" Jul 31 18:44:40.692765 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:40.692733 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c5dadcb-6709-4bad-9a34-d4bdfe44fce1-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-kbx2l\" (UID: \"7c5dadcb-6709-4bad-9a34-d4bdfe44fce1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-kbx2l" Jul 31 18:44:40.692920 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:40.692823 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-registry-tls\") pod \"image-registry-6bf57fb7b8-l6h5t\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" Jul 31 18:44:40.692920 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:40.692879 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 31 18:44:40.692920 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:40.692910 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 18:44:40.692920 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:40.692920 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bf57fb7b8-l6h5t: secret "image-registry-tls" not found Jul 31 18:44:40.693038 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:40.692940 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c5dadcb-6709-4bad-9a34-d4bdfe44fce1-samples-operator-tls podName:7c5dadcb-6709-4bad-9a34-d4bdfe44fce1 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:48.692926263 +0000 UTC m=+65.330916041 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7c5dadcb-6709-4bad-9a34-d4bdfe44fce1-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-kbx2l" (UID: "7c5dadcb-6709-4bad-9a34-d4bdfe44fce1") : secret "samples-operator-tls" not found Jul 31 18:44:40.693038 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:40.692957 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-registry-tls podName:cdc4eaf7-429f-4fc0-be49-1c4913cfdb85 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:48.692945586 +0000 UTC m=+65.330935394 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-registry-tls") pod "image-registry-6bf57fb7b8-l6h5t" (UID: "cdc4eaf7-429f-4fc0-be49-1c4913cfdb85") : secret "image-registry-tls" not found Jul 31 18:44:40.793943 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:40.793906 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8c11319-084e-4318-9e92-5a4a68d8c916-metrics-certs\") pod \"router-default-685dfffc96-vrmgp\" (UID: \"a8c11319-084e-4318-9e92-5a4a68d8c916\") " pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:44:40.794104 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:40.793976 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8c11319-084e-4318-9e92-5a4a68d8c916-service-ca-bundle\") pod \"router-default-685dfffc96-vrmgp\" (UID: \"a8c11319-084e-4318-9e92-5a4a68d8c916\") " pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:44:40.794104 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:40.794051 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 31 18:44:40.794104 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:40.794084 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a8c11319-084e-4318-9e92-5a4a68d8c916-service-ca-bundle podName:a8c11319-084e-4318-9e92-5a4a68d8c916 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:48.7940691 +0000 UTC m=+65.432058878 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a8c11319-084e-4318-9e92-5a4a68d8c916-service-ca-bundle") pod "router-default-685dfffc96-vrmgp" (UID: "a8c11319-084e-4318-9e92-5a4a68d8c916") : configmap references non-existent config key: service-ca.crt Jul 31 18:44:40.794104 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:40.794106 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8c11319-084e-4318-9e92-5a4a68d8c916-metrics-certs podName:a8c11319-084e-4318-9e92-5a4a68d8c916 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:48.79409935 +0000 UTC m=+65.432089128 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8c11319-084e-4318-9e92-5a4a68d8c916-metrics-certs") pod "router-default-685dfffc96-vrmgp" (UID: "a8c11319-084e-4318-9e92-5a4a68d8c916") : secret "router-metrics-certs-default" not found Jul 31 18:44:40.799799 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:40.799773 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:40.901348 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:40.901319 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:41.351075 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:41.351051 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9kv6f_9bd58afe-8ba9-4f95-ae27-c97586229a7a/serve-healthcheck-canary/0.log" Jul 31 18:44:41.552137 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:41.552095 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:41.689387 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:41.689302 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:41.702509 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:41.702484 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e664160-fd29-4f67-b32e-8ae37b2641be-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vrdbp\" (UID: \"0e664160-fd29-4f67-b32e-8ae37b2641be\") " pod="openshift-insights/insights-runtime-extractor-vrdbp" Jul 31 18:44:41.702627 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:41.702594 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 18:44:41.702627 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:41.702613 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e664160-fd29-4f67-b32e-8ae37b2641be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vrdbp\" (UID: \"0e664160-fd29-4f67-b32e-8ae37b2641be\") " pod="openshift-insights/insights-runtime-extractor-vrdbp" Jul 31 18:44:41.702735 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:41.702647 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e664160-fd29-4f67-b32e-8ae37b2641be-kube-rbac-proxy-cm podName:0e664160-fd29-4f67-b32e-8ae37b2641be nodeName:}" failed. No retries permitted until 2026-07-31 18:44:45.702633308 +0000 UTC m=+62.340623086 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0e664160-fd29-4f67-b32e-8ae37b2641be-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vrdbp" (UID: "0e664160-fd29-4f67-b32e-8ae37b2641be") : configmap "kube-rbac-proxy" not found Jul 31 18:44:41.702735 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:41.702715 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 18:44:41.702831 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:41.702774 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e664160-fd29-4f67-b32e-8ae37b2641be-insights-runtime-extractor-tls podName:0e664160-fd29-4f67-b32e-8ae37b2641be nodeName:}" failed. No retries permitted until 2026-07-31 18:44:45.702758189 +0000 UTC m=+62.340748036 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e664160-fd29-4f67-b32e-8ae37b2641be-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vrdbp" (UID: "0e664160-fd29-4f67-b32e-8ae37b2641be") : secret "insights-runtime-extractor-tls" not found Jul 31 18:44:42.525739 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:42.525689 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:44.147221 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:44.147194 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fxm9n" Jul 31 18:44:45.739630 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:45.739594 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e664160-fd29-4f67-b32e-8ae37b2641be-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vrdbp\" (UID: \"0e664160-fd29-4f67-b32e-8ae37b2641be\") " pod="openshift-insights/insights-runtime-extractor-vrdbp" Jul 31 18:44:45.740011 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:45.739720 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 18:44:45.740011 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:45.739764 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e664160-fd29-4f67-b32e-8ae37b2641be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vrdbp\" (UID: \"0e664160-fd29-4f67-b32e-8ae37b2641be\") " pod="openshift-insights/insights-runtime-extractor-vrdbp" Jul 31 18:44:45.740011 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:45.739782 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e664160-fd29-4f67-b32e-8ae37b2641be-kube-rbac-proxy-cm podName:0e664160-fd29-4f67-b32e-8ae37b2641be nodeName:}" failed. No retries permitted until 2026-07-31 18:44:53.739765536 +0000 UTC m=+70.377755313 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0e664160-fd29-4f67-b32e-8ae37b2641be-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vrdbp" (UID: "0e664160-fd29-4f67-b32e-8ae37b2641be") : configmap "kube-rbac-proxy" not found Jul 31 18:44:45.740011 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:45.739840 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 18:44:45.740011 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:45.739888 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e664160-fd29-4f67-b32e-8ae37b2641be-insights-runtime-extractor-tls podName:0e664160-fd29-4f67-b32e-8ae37b2641be nodeName:}" failed. No retries permitted until 2026-07-31 18:44:53.739877584 +0000 UTC m=+70.377867362 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e664160-fd29-4f67-b32e-8ae37b2641be-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vrdbp" (UID: "0e664160-fd29-4f67-b32e-8ae37b2641be") : secret "insights-runtime-extractor-tls" not found Jul 31 18:44:46.249870 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:46.249830 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:46.675728 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:46.675623 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:46.777395 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:46.777364 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:46.893452 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:46.893414 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:47.072582 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:47.072534 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:47.334327 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:47.334232 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:47.455190 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:47.455158 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:47.970222 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:47.970188 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:48.564596 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:48.564553 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/45c0e651-aba9-4741-84db-8a06cd82c46e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-m76n7\" (UID: \"45c0e651-aba9-4741-84db-8a06cd82c46e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m76n7" Jul 31 18:44:48.564768 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:48.564680 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 31 18:44:48.564768 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:48.564745 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45c0e651-aba9-4741-84db-8a06cd82c46e-cluster-monitoring-operator-tls podName:45c0e651-aba9-4741-84db-8a06cd82c46e nodeName:}" failed. No retries permitted until 2026-07-31 18:45:04.564732425 +0000 UTC m=+81.202722203 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/45c0e651-aba9-4741-84db-8a06cd82c46e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-m76n7" (UID: "45c0e651-aba9-4741-84db-8a06cd82c46e") : secret "cluster-monitoring-operator-tls" not found Jul 31 18:44:48.587408 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:48.587375 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:48.765738 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:48.765703 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c5dadcb-6709-4bad-9a34-d4bdfe44fce1-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-kbx2l\" (UID: \"7c5dadcb-6709-4bad-9a34-d4bdfe44fce1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-kbx2l" Jul 31 18:44:48.765908 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:48.765807 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-registry-tls\") pod \"image-registry-6bf57fb7b8-l6h5t\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" Jul 31 18:44:48.765908 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:48.765896 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 18:44:48.765908 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:48.765907 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bf57fb7b8-l6h5t: secret "image-registry-tls" not found Jul 31 18:44:48.766020 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:48.765954 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-registry-tls podName:cdc4eaf7-429f-4fc0-be49-1c4913cfdb85 nodeName:}" failed. No retries permitted until 2026-07-31 18:45:04.765942724 +0000 UTC m=+81.403932502 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-registry-tls") pod "image-registry-6bf57fb7b8-l6h5t" (UID: "cdc4eaf7-429f-4fc0-be49-1c4913cfdb85") : secret "image-registry-tls" not found Jul 31 18:44:48.768080 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:48.768048 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c5dadcb-6709-4bad-9a34-d4bdfe44fce1-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-kbx2l\" (UID: \"7c5dadcb-6709-4bad-9a34-d4bdfe44fce1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-kbx2l" Jul 31 18:44:48.866203 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:48.866123 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8c11319-084e-4318-9e92-5a4a68d8c916-service-ca-bundle\") pod \"router-default-685dfffc96-vrmgp\" (UID: \"a8c11319-084e-4318-9e92-5a4a68d8c916\") " pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:44:48.866352 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:48.866205 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8c11319-084e-4318-9e92-5a4a68d8c916-metrics-certs\") pod \"router-default-685dfffc96-vrmgp\" (UID: \"a8c11319-084e-4318-9e92-5a4a68d8c916\") " pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:44:48.866352 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:48.866299 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a8c11319-084e-4318-9e92-5a4a68d8c916-service-ca-bundle podName:a8c11319-084e-4318-9e92-5a4a68d8c916 nodeName:}" failed. No retries permitted until 2026-07-31 18:45:04.866283531 +0000 UTC m=+81.504273309 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a8c11319-084e-4318-9e92-5a4a68d8c916-service-ca-bundle") pod "router-default-685dfffc96-vrmgp" (UID: "a8c11319-084e-4318-9e92-5a4a68d8c916") : configmap references non-existent config key: service-ca.crt Jul 31 18:44:48.866352 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:48.866313 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 31 18:44:48.866352 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:48.866349 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8c11319-084e-4318-9e92-5a4a68d8c916-metrics-certs podName:a8c11319-084e-4318-9e92-5a4a68d8c916 nodeName:}" failed. No retries permitted until 2026-07-31 18:45:04.866339922 +0000 UTC m=+81.504329701 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8c11319-084e-4318-9e92-5a4a68d8c916-metrics-certs") pod "router-default-685dfffc96-vrmgp" (UID: "a8c11319-084e-4318-9e92-5a4a68d8c916") : secret "router-metrics-certs-default" not found Jul 31 18:44:48.974159 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:48.974118 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:49.057248 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:49.057210 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-pxd4x\"" Jul 31 18:44:49.065176 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:49.065157 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-kbx2l" Jul 31 18:44:49.194307 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:49.194268 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-kbx2l"] Jul 31 18:44:49.235628 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:49.235604 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-kbx2l" event={"ID":"7c5dadcb-6709-4bad-9a34-d4bdfe44fce1","Type":"ContainerStarted","Data":"7debe19a62704be5402b5997866b560625fc15d1f7deb58d5ca188b2f13ddead"} Jul 31 18:44:49.368974 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:49.368937 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:49.470792 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:49.470703 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:49.571148 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:49.571107 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee8aa68f-72ff-4118-ad38-4da03b305feb-metrics-certs\") pod \"network-metrics-daemon-h8n5q\" (UID: \"ee8aa68f-72ff-4118-ad38-4da03b305feb\") " pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:44:49.571357 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:49.571176 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8223854-5549-4701-b1ce-506ed19d8abe-original-pull-secret\") pod \"global-pull-secret-syncer-2rkhw\" (UID: \"e8223854-5549-4701-b1ce-506ed19d8abe\") " pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:44:49.573978 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:49.573960 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 31 18:44:49.574621 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:49.574607 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 31 18:44:49.581935 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:49.581911 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 31 18:44:49.582030 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:49.581979 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee8aa68f-72ff-4118-ad38-4da03b305feb-metrics-certs podName:ee8aa68f-72ff-4118-ad38-4da03b305feb nodeName:}" failed. No retries permitted until 2026-07-31 18:45:53.581961501 +0000 UTC m=+130.219951279 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee8aa68f-72ff-4118-ad38-4da03b305feb-metrics-certs") pod "network-metrics-daemon-h8n5q" (UID: "ee8aa68f-72ff-4118-ad38-4da03b305feb") : secret "metrics-daemon-secret" not found Jul 31 18:44:49.584094 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:49.584074 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8223854-5549-4701-b1ce-506ed19d8abe-original-pull-secret\") pod \"global-pull-secret-syncer-2rkhw\" (UID: \"e8223854-5549-4701-b1ce-506ed19d8abe\") " pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:44:49.590377 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:49.590354 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:49.672371 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:49.672334 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k697\" (UniqueName: \"kubernetes.io/projected/0914ed48-d402-4d77-a2e7-ca12693fd0f0-kube-api-access-2k697\") pod \"network-check-target-xqlm2\" (UID: \"0914ed48-d402-4d77-a2e7-ca12693fd0f0\") " pod="openshift-network-diagnostics/network-check-target-xqlm2" Jul 31 18:44:49.674846 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:49.674828 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 31 18:44:49.685469 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:49.685446 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 31 18:44:49.696537 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:49.696515 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k697\" (UniqueName: \"kubernetes.io/projected/0914ed48-d402-4d77-a2e7-ca12693fd0f0-kube-api-access-2k697\") pod \"network-check-target-xqlm2\" (UID: \"0914ed48-d402-4d77-a2e7-ca12693fd0f0\") " pod="openshift-network-diagnostics/network-check-target-xqlm2" Jul 31 18:44:49.807963 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:49.807931 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkhw" Jul 31 18:44:49.816163 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:49.816137 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-99tgj\"" Jul 31 18:44:49.823960 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:49.823939 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqlm2" Jul 31 18:44:49.949664 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:49.949627 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2rkhw"] Jul 31 18:44:49.953432 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:44:49.953402 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8223854_5549_4701_b1ce_506ed19d8abe.slice/crio-d0b8c22619e0f2ddbce9e7164961440a3979bdbc6802d62259df9fca95f839cb WatchSource:0}: Error finding container d0b8c22619e0f2ddbce9e7164961440a3979bdbc6802d62259df9fca95f839cb: Status 404 returned error can't find the container with id d0b8c22619e0f2ddbce9e7164961440a3979bdbc6802d62259df9fca95f839cb Jul 31 18:44:49.967749 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:49.967721 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xqlm2"] Jul 31 18:44:49.971140 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:44:49.971111 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0914ed48_d402_4d77_a2e7_ca12693fd0f0.slice/crio-1e21a1dd40e8826208557312b1fe6dbb21c4099895827b776dfd309ec8e20144 WatchSource:0}: Error finding container 1e21a1dd40e8826208557312b1fe6dbb21c4099895827b776dfd309ec8e20144: Status 404 returned error can't find the container with id 1e21a1dd40e8826208557312b1fe6dbb21c4099895827b776dfd309ec8e20144 Jul 31 18:44:49.973177 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:49.973144 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:50.240160 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:50.240074 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xqlm2" event={"ID":"0914ed48-d402-4d77-a2e7-ca12693fd0f0","Type":"ContainerStarted","Data":"1e21a1dd40e8826208557312b1fe6dbb21c4099895827b776dfd309ec8e20144"} Jul 31 18:44:50.241312 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:50.241281 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2rkhw" event={"ID":"e8223854-5549-4701-b1ce-506ed19d8abe","Type":"ContainerStarted","Data":"d0b8c22619e0f2ddbce9e7164961440a3979bdbc6802d62259df9fca95f839cb"} Jul 31 18:44:50.357018 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:50.356986 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:50.491957 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:50.491819 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:50.639714 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:50.639670 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:50.781506 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:50.781472 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98250d35-1d7d-4f70-8350-b7865456ae46-metrics-tls\") pod \"dns-default-dgmn6\" (UID: \"98250d35-1d7d-4f70-8350-b7865456ae46\") " pod="openshift-dns/dns-default-dgmn6" Jul 31 18:44:50.785592 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:50.785531 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98250d35-1d7d-4f70-8350-b7865456ae46-metrics-tls\") pod \"dns-default-dgmn6\" (UID: \"98250d35-1d7d-4f70-8350-b7865456ae46\") " pod="openshift-dns/dns-default-dgmn6" Jul 31 18:44:50.995114 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:50.995075 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v8wjd\"" Jul 31 18:44:51.000525 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:51.000499 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dgmn6" Jul 31 18:44:51.018174 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:51.018118 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:51.182355 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:51.182256 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:51.374891 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:51.374839 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:51.446683 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:51.446635 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dgmn6"] Jul 31 18:44:51.450828 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:44:51.450794 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98250d35_1d7d_4f70_8350_b7865456ae46.slice/crio-9c14f15bca2b58e32940f993b8ce758fce439fdd478bd8afda8dcf6b8d6dc779 WatchSource:0}: Error finding container 9c14f15bca2b58e32940f993b8ce758fce439fdd478bd8afda8dcf6b8d6dc779: Status 404 returned error can't find the container with id 9c14f15bca2b58e32940f993b8ce758fce439fdd478bd8afda8dcf6b8d6dc779 Jul 31 18:44:51.478655 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:51.478619 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:51.584148 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:51.584068 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:52.028724 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:52.028686 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:52.113724 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:52.113631 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:52.248881 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:52.248799 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dgmn6" event={"ID":"98250d35-1d7d-4f70-8350-b7865456ae46","Type":"ContainerStarted","Data":"9c14f15bca2b58e32940f993b8ce758fce439fdd478bd8afda8dcf6b8d6dc779"} Jul 31 18:44:52.251068 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:52.251026 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-kbx2l" event={"ID":"7c5dadcb-6709-4bad-9a34-d4bdfe44fce1","Type":"ContainerStarted","Data":"572346ebad5bd8103ae4303d888cde26dd59017af6e99311ee176ba7ba2d6101"} Jul 31 18:44:52.251211 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:52.251082 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-kbx2l" event={"ID":"7c5dadcb-6709-4bad-9a34-d4bdfe44fce1","Type":"ContainerStarted","Data":"5dded66c9cc8186da639349bd9a960aedb128e1e1b2f434407db0383c7999b1e"} Jul 31 18:44:52.327288 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:52.327215 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:52.406419 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:52.406337 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:52.764199 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:52.764077 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:52.873711 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:52.873673 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:53.245824 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:53.245739 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:53.769067 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:53.769034 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:53.813915 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:53.813856 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e664160-fd29-4f67-b32e-8ae37b2641be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vrdbp\" (UID: \"0e664160-fd29-4f67-b32e-8ae37b2641be\") " pod="openshift-insights/insights-runtime-extractor-vrdbp" Jul 31 18:44:53.814079 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:53.813969 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e664160-fd29-4f67-b32e-8ae37b2641be-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vrdbp\" (UID: \"0e664160-fd29-4f67-b32e-8ae37b2641be\") " pod="openshift-insights/insights-runtime-extractor-vrdbp" Jul 31 18:44:53.814079 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:53.813973 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 18:44:53.814200 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:44:53.814093 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e664160-fd29-4f67-b32e-8ae37b2641be-insights-runtime-extractor-tls podName:0e664160-fd29-4f67-b32e-8ae37b2641be nodeName:}" failed. No retries permitted until 2026-07-31 18:45:09.814073927 +0000 UTC m=+86.452063707 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e664160-fd29-4f67-b32e-8ae37b2641be-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vrdbp" (UID: "0e664160-fd29-4f67-b32e-8ae37b2641be") : secret "insights-runtime-extractor-tls" not found Jul 31 18:44:53.814566 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:53.814539 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e664160-fd29-4f67-b32e-8ae37b2641be-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vrdbp\" (UID: \"0e664160-fd29-4f67-b32e-8ae37b2641be\") " pod="openshift-insights/insights-runtime-extractor-vrdbp" Jul 31 18:44:53.992543 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:53.992502 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:54.356602 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:54.356556 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:54.564142 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:54.564100 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:54.764211 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:54.764180 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:54.981537 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:54.981502 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:55.634295 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:55.634263 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:55.737967 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:55.737919 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:55.973617 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:55.973512 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:56.263660 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:56.263622 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xqlm2" event={"ID":"0914ed48-d402-4d77-a2e7-ca12693fd0f0","Type":"ContainerStarted","Data":"d6a002d8fe0030dbb551c543912ca58de0ca739736d7704b4064ec0b053e9356"} Jul 31 18:44:56.264105 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:56.263695 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xqlm2" Jul 31 18:44:56.265051 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:56.265031 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2rkhw" event={"ID":"e8223854-5549-4701-b1ce-506ed19d8abe","Type":"ContainerStarted","Data":"9e4b8095a6632397b2fd780d58d31d50a4ca6789c064df29d2ff35501acdb970"} Jul 31 18:44:56.266596 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:56.266573 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dgmn6" event={"ID":"98250d35-1d7d-4f70-8350-b7865456ae46","Type":"ContainerStarted","Data":"6b8727c89ed70aa6a0edf5daaf7951394fe735f00d2a351eaaf736b8bef6f2f7"} Jul 31 18:44:56.266718 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:56.266619 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dgmn6" event={"ID":"98250d35-1d7d-4f70-8350-b7865456ae46","Type":"ContainerStarted","Data":"bce292676bbc3dee56c01c915a2882658a057e32386b0895b9a1f28cdc888e60"} Jul 31 18:44:56.266718 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:56.266653 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dgmn6" Jul 31 18:44:56.282751 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:56.282714 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-kbx2l" podStartSLOduration=21.875514084 podStartE2EDuration="24.282702598s" totalImagesPullingTime="2.407188514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:44:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:52.276635554 +0000 UTC m=+68.914625358" watchObservedRunningTime="2026-07-31 18:44:56.282702598 +0000 UTC m=+72.920692455" Jul 31 18:44:56.282901 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:56.282882 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xqlm2" podStartSLOduration=67.060321314 podStartE2EDuration="1m12.282877256s" totalImagesPullingTime="5.222555942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:56.282207484 +0000 UTC m=+72.920197285" watchObservedRunningTime="2026-07-31 18:44:56.282877256 +0000 UTC m=+72.920867049" Jul 31 18:44:56.310561 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:56.310523 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dgmn6" podStartSLOduration=34.743626693 podStartE2EDuration="38.310511265s" totalImagesPullingTime="3.566884572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:56.309897776 +0000 UTC m=+72.947887577" watchObservedRunningTime="2026-07-31 18:44:56.310511265 +0000 UTC m=+72.948501099" Jul 31 18:44:56.340608 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:56.340566 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2rkhw" podStartSLOduration=67.088990383 podStartE2EDuration="1m12.340554535s" totalImagesPullingTime="5.251564152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:56.340464802 +0000 UTC m=+72.978454624" watchObservedRunningTime="2026-07-31 18:44:56.340554535 +0000 UTC m=+72.978544335" Jul 31 18:44:56.386118 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:56.386089 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:56.561557 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:56.561477 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:56.761420 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:56.761384 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:57.170676 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:57.170645 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:57.385139 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:57.385110 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:57.871789 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:57.871753 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:57.973731 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:57.973696 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:58.049233 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:58.049199 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:58.178228 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:58.178145 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:58.546861 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:58.546822 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:58.647495 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:58.647457 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:59.004430 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:59.004400 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:59.175775 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:59.175747 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:59.575679 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:59.575653 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:59.760907 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:59.760877 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:59.860192 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:59.860096 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:44:59.981103 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:44:59.981073 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:00.048889 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:00.048836 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:01.984125 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:01.984089 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-6fcww"] Jul 31 18:45:01.989294 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:01.989262 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-6fcww" Jul 31 18:45:01.993136 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:01.993113 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 31 18:45:01.993719 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:01.993685 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-c57gd\"" Jul 31 18:45:01.993846 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:01.993719 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 31 18:45:02.000935 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.000912 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-6fcww"] Jul 31 18:45:02.013454 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.013425 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b6f988b74-gvnd4"] Jul 31 18:45:02.016966 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.016943 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6bf57fb7b8-l6h5t"] Jul 31 18:45:02.017106 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.017089 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b6f988b74-gvnd4" Jul 31 18:45:02.017265 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:45:02.017220 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" podUID="cdc4eaf7-429f-4fc0-be49-1c4913cfdb85" Jul 31 18:45:02.019320 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.019296 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d86d85884-shqbj"] Jul 31 18:45:02.022607 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.022589 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d86d85884-shqbj" Jul 31 18:45:02.023707 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.023690 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 31 18:45:02.023804 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.023779 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 31 18:45:02.024295 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.024227 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-l75lv\"" Jul 31 18:45:02.024367 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.024353 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 31 18:45:02.027404 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.027386 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 31 18:45:02.029491 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.029468 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 31 18:45:02.041404 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.041379 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b6f988b74-gvnd4"] Jul 31 18:45:02.050790 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.050768 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d86d85884-shqbj"] Jul 31 18:45:02.076285 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.076255 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:02.085358 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.085330 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5efb0219-e98d-45a6-ad94-122359fc7cda-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-b6f988b74-gvnd4\" (UID: \"5efb0219-e98d-45a6-ad94-122359fc7cda\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b6f988b74-gvnd4" Jul 31 18:45:02.085493 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.085362 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0b029477-8e2e-46ba-8209-d3cd2f49f582-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-6fcww\" (UID: \"0b029477-8e2e-46ba-8209-d3cd2f49f582\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-6fcww" Jul 31 18:45:02.085493 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.085465 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0b029477-8e2e-46ba-8209-d3cd2f49f582-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-6fcww\" (UID: \"0b029477-8e2e-46ba-8209-d3cd2f49f582\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-6fcww" Jul 31 18:45:02.085565 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.085534 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2s55\" (UniqueName: \"kubernetes.io/projected/5efb0219-e98d-45a6-ad94-122359fc7cda-kube-api-access-x2s55\") pod \"managed-serviceaccount-addon-agent-b6f988b74-gvnd4\" (UID: \"5efb0219-e98d-45a6-ad94-122359fc7cda\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b6f988b74-gvnd4" Jul 31 18:45:02.088746 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.088725 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-77c974ffcf-hb2hk"] Jul 31 18:45:02.092330 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.092306 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-hb2hk" Jul 31 18:45:02.094466 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.094444 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 31 18:45:02.094825 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.094806 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 31 18:45:02.094925 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.094808 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-rtsfg\"" Jul 31 18:45:02.096165 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.096140 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b4695766f-dvcj6"] Jul 31 18:45:02.099407 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.099388 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b4695766f-dvcj6" Jul 31 18:45:02.107548 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.107527 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-hb2hk"] Jul 31 18:45:02.131487 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.131461 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b4695766f-dvcj6"] Jul 31 18:45:02.186438 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.186410 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9c6956f-b84a-41c3-a728-7c0f0895a651-tmp\") pod \"klusterlet-addon-workmgr-6d86d85884-shqbj\" (UID: \"a9c6956f-b84a-41c3-a728-7c0f0895a651\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d86d85884-shqbj" Jul 31 18:45:02.186438 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.186441 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3bea09ae-5eb5-4b26-a6d3-793ed2defb45-ca-trust-extracted\") pod \"image-registry-b4695766f-dvcj6\" (UID: \"3bea09ae-5eb5-4b26-a6d3-793ed2defb45\") " pod="openshift-image-registry/image-registry-b4695766f-dvcj6" Jul 31 18:45:02.186607 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.186462 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0b029477-8e2e-46ba-8209-d3cd2f49f582-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-6fcww\" (UID: \"0b029477-8e2e-46ba-8209-d3cd2f49f582\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-6fcww" Jul 31 18:45:02.186607 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.186552 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3bea09ae-5eb5-4b26-a6d3-793ed2defb45-installation-pull-secrets\") pod \"image-registry-b4695766f-dvcj6\" (UID: \"3bea09ae-5eb5-4b26-a6d3-793ed2defb45\") " pod="openshift-image-registry/image-registry-b4695766f-dvcj6" Jul 31 18:45:02.186815 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.186792 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2s55\" (UniqueName: \"kubernetes.io/projected/5efb0219-e98d-45a6-ad94-122359fc7cda-kube-api-access-x2s55\") pod \"managed-serviceaccount-addon-agent-b6f988b74-gvnd4\" (UID: \"5efb0219-e98d-45a6-ad94-122359fc7cda\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b6f988b74-gvnd4" Jul 31 18:45:02.186876 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.186838 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjjzn\" (UniqueName: \"kubernetes.io/projected/a9c6956f-b84a-41c3-a728-7c0f0895a651-kube-api-access-gjjzn\") pod \"klusterlet-addon-workmgr-6d86d85884-shqbj\" (UID: \"a9c6956f-b84a-41c3-a728-7c0f0895a651\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d86d85884-shqbj" Jul 31 18:45:02.186924 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.186882 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xj5s\" (UniqueName: \"kubernetes.io/projected/3bea09ae-5eb5-4b26-a6d3-793ed2defb45-kube-api-access-8xj5s\") pod \"image-registry-b4695766f-dvcj6\" (UID: \"3bea09ae-5eb5-4b26-a6d3-793ed2defb45\") " pod="openshift-image-registry/image-registry-b4695766f-dvcj6" Jul 31 18:45:02.186924 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.186916 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3bea09ae-5eb5-4b26-a6d3-793ed2defb45-bound-sa-token\") pod \"image-registry-b4695766f-dvcj6\" (UID: \"3bea09ae-5eb5-4b26-a6d3-793ed2defb45\") " pod="openshift-image-registry/image-registry-b4695766f-dvcj6" Jul 31 18:45:02.187144 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.186943 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3bea09ae-5eb5-4b26-a6d3-793ed2defb45-registry-certificates\") pod \"image-registry-b4695766f-dvcj6\" (UID: \"3bea09ae-5eb5-4b26-a6d3-793ed2defb45\") " pod="openshift-image-registry/image-registry-b4695766f-dvcj6" Jul 31 18:45:02.187144 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.186971 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a9c6956f-b84a-41c3-a728-7c0f0895a651-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d86d85884-shqbj\" (UID: \"a9c6956f-b84a-41c3-a728-7c0f0895a651\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d86d85884-shqbj" Jul 31 18:45:02.187144 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.187063 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5efb0219-e98d-45a6-ad94-122359fc7cda-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-b6f988b74-gvnd4\" (UID: \"5efb0219-e98d-45a6-ad94-122359fc7cda\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b6f988b74-gvnd4" Jul 31 18:45:02.187144 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.187106 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0b029477-8e2e-46ba-8209-d3cd2f49f582-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-6fcww\" (UID: \"0b029477-8e2e-46ba-8209-d3cd2f49f582\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-6fcww" Jul 31 18:45:02.187144 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.187134 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3bea09ae-5eb5-4b26-a6d3-793ed2defb45-registry-tls\") pod \"image-registry-b4695766f-dvcj6\" (UID: \"3bea09ae-5eb5-4b26-a6d3-793ed2defb45\") " pod="openshift-image-registry/image-registry-b4695766f-dvcj6" Jul 31 18:45:02.187405 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.187161 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws5xv\" (UniqueName: \"kubernetes.io/projected/e3d33d0d-4656-4016-be66-99d9aa6e0994-kube-api-access-ws5xv\") pod \"downloads-77c974ffcf-hb2hk\" (UID: \"e3d33d0d-4656-4016-be66-99d9aa6e0994\") " pod="openshift-console/downloads-77c974ffcf-hb2hk" Jul 31 18:45:02.187405 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.187257 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3bea09ae-5eb5-4b26-a6d3-793ed2defb45-trusted-ca\") pod \"image-registry-b4695766f-dvcj6\" (UID: \"3bea09ae-5eb5-4b26-a6d3-793ed2defb45\") " pod="openshift-image-registry/image-registry-b4695766f-dvcj6" Jul 31 18:45:02.187405 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.187299 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3bea09ae-5eb5-4b26-a6d3-793ed2defb45-image-registry-private-configuration\") pod \"image-registry-b4695766f-dvcj6\" (UID: \"3bea09ae-5eb5-4b26-a6d3-793ed2defb45\") " pod="openshift-image-registry/image-registry-b4695766f-dvcj6" Jul 31 18:45:02.187889 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.187867 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0b029477-8e2e-46ba-8209-d3cd2f49f582-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-6fcww\" (UID: \"0b029477-8e2e-46ba-8209-d3cd2f49f582\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-6fcww" Jul 31 18:45:02.190031 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.190010 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0b029477-8e2e-46ba-8209-d3cd2f49f582-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-6fcww\" (UID: \"0b029477-8e2e-46ba-8209-d3cd2f49f582\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-6fcww" Jul 31 18:45:02.190197 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.190178 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5efb0219-e98d-45a6-ad94-122359fc7cda-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-b6f988b74-gvnd4\" (UID: \"5efb0219-e98d-45a6-ad94-122359fc7cda\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b6f988b74-gvnd4" Jul 31 18:45:02.206071 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.206048 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2s55\" (UniqueName: \"kubernetes.io/projected/5efb0219-e98d-45a6-ad94-122359fc7cda-kube-api-access-x2s55\") pod \"managed-serviceaccount-addon-agent-b6f988b74-gvnd4\" (UID: \"5efb0219-e98d-45a6-ad94-122359fc7cda\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b6f988b74-gvnd4" Jul 31 18:45:02.283694 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.283668 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" Jul 31 18:45:02.287666 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.287639 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3bea09ae-5eb5-4b26-a6d3-793ed2defb45-trusted-ca\") pod \"image-registry-b4695766f-dvcj6\" (UID: \"3bea09ae-5eb5-4b26-a6d3-793ed2defb45\") " pod="openshift-image-registry/image-registry-b4695766f-dvcj6" Jul 31 18:45:02.287801 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.287681 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3bea09ae-5eb5-4b26-a6d3-793ed2defb45-image-registry-private-configuration\") pod \"image-registry-b4695766f-dvcj6\" (UID: \"3bea09ae-5eb5-4b26-a6d3-793ed2defb45\") " pod="openshift-image-registry/image-registry-b4695766f-dvcj6" Jul 31 18:45:02.287801 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.287734 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9c6956f-b84a-41c3-a728-7c0f0895a651-tmp\") pod \"klusterlet-addon-workmgr-6d86d85884-shqbj\" (UID: \"a9c6956f-b84a-41c3-a728-7c0f0895a651\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d86d85884-shqbj" Jul 31 18:45:02.287801 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.287754 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3bea09ae-5eb5-4b26-a6d3-793ed2defb45-ca-trust-extracted\") pod \"image-registry-b4695766f-dvcj6\" (UID: \"3bea09ae-5eb5-4b26-a6d3-793ed2defb45\") " pod="openshift-image-registry/image-registry-b4695766f-dvcj6" Jul 31 18:45:02.287801 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.287796 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3bea09ae-5eb5-4b26-a6d3-793ed2defb45-installation-pull-secrets\") pod \"image-registry-b4695766f-dvcj6\" (UID: \"3bea09ae-5eb5-4b26-a6d3-793ed2defb45\") " pod="openshift-image-registry/image-registry-b4695766f-dvcj6" Jul 31 18:45:02.288015 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.287851 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjjzn\" (UniqueName: \"kubernetes.io/projected/a9c6956f-b84a-41c3-a728-7c0f0895a651-kube-api-access-gjjzn\") pod \"klusterlet-addon-workmgr-6d86d85884-shqbj\" (UID: \"a9c6956f-b84a-41c3-a728-7c0f0895a651\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d86d85884-shqbj" Jul 31 18:45:02.288015 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.287893 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xj5s\" (UniqueName: \"kubernetes.io/projected/3bea09ae-5eb5-4b26-a6d3-793ed2defb45-kube-api-access-8xj5s\") pod \"image-registry-b4695766f-dvcj6\" (UID: \"3bea09ae-5eb5-4b26-a6d3-793ed2defb45\") " pod="openshift-image-registry/image-registry-b4695766f-dvcj6" Jul 31 18:45:02.288015 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.287923 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3bea09ae-5eb5-4b26-a6d3-793ed2defb45-bound-sa-token\") pod \"image-registry-b4695766f-dvcj6\" (UID: \"3bea09ae-5eb5-4b26-a6d3-793ed2defb45\") " pod="openshift-image-registry/image-registry-b4695766f-dvcj6" Jul 31 18:45:02.288015 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.287947 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3bea09ae-5eb5-4b26-a6d3-793ed2defb45-registry-certificates\") pod \"image-registry-b4695766f-dvcj6\" (UID: \"3bea09ae-5eb5-4b26-a6d3-793ed2defb45\") " pod="openshift-image-registry/image-registry-b4695766f-dvcj6" Jul 31 18:45:02.288015 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.287976 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a9c6956f-b84a-41c3-a728-7c0f0895a651-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d86d85884-shqbj\" (UID: \"a9c6956f-b84a-41c3-a728-7c0f0895a651\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d86d85884-shqbj" Jul 31 18:45:02.288287 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.288035 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3bea09ae-5eb5-4b26-a6d3-793ed2defb45-registry-tls\") pod \"image-registry-b4695766f-dvcj6\" (UID: \"3bea09ae-5eb5-4b26-a6d3-793ed2defb45\") " pod="openshift-image-registry/image-registry-b4695766f-dvcj6" Jul 31 18:45:02.288287 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.288063 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws5xv\" (UniqueName: \"kubernetes.io/projected/e3d33d0d-4656-4016-be66-99d9aa6e0994-kube-api-access-ws5xv\") pod \"downloads-77c974ffcf-hb2hk\" (UID: \"e3d33d0d-4656-4016-be66-99d9aa6e0994\") " pod="openshift-console/downloads-77c974ffcf-hb2hk" Jul 31 18:45:02.288287 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.288172 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9c6956f-b84a-41c3-a728-7c0f0895a651-tmp\") pod \"klusterlet-addon-workmgr-6d86d85884-shqbj\" (UID: \"a9c6956f-b84a-41c3-a728-7c0f0895a651\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d86d85884-shqbj" Jul 31 18:45:02.288662 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.288593 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3bea09ae-5eb5-4b26-a6d3-793ed2defb45-ca-trust-extracted\") pod \"image-registry-b4695766f-dvcj6\" (UID: \"3bea09ae-5eb5-4b26-a6d3-793ed2defb45\") " pod="openshift-image-registry/image-registry-b4695766f-dvcj6" Jul 31 18:45:02.289313 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.289287 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3bea09ae-5eb5-4b26-a6d3-793ed2defb45-registry-certificates\") pod \"image-registry-b4695766f-dvcj6\" (UID: \"3bea09ae-5eb5-4b26-a6d3-793ed2defb45\") " pod="openshift-image-registry/image-registry-b4695766f-dvcj6" Jul 31 18:45:02.289666 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.289640 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3bea09ae-5eb5-4b26-a6d3-793ed2defb45-trusted-ca\") pod \"image-registry-b4695766f-dvcj6\" (UID: \"3bea09ae-5eb5-4b26-a6d3-793ed2defb45\") " pod="openshift-image-registry/image-registry-b4695766f-dvcj6" Jul 31 18:45:02.290623 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.290599 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3bea09ae-5eb5-4b26-a6d3-793ed2defb45-installation-pull-secrets\") pod \"image-registry-b4695766f-dvcj6\" (UID: \"3bea09ae-5eb5-4b26-a6d3-793ed2defb45\") " pod="openshift-image-registry/image-registry-b4695766f-dvcj6" Jul 31 18:45:02.290724 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.290701 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3bea09ae-5eb5-4b26-a6d3-793ed2defb45-image-registry-private-configuration\") pod \"image-registry-b4695766f-dvcj6\" (UID: \"3bea09ae-5eb5-4b26-a6d3-793ed2defb45\") " pod="openshift-image-registry/image-registry-b4695766f-dvcj6" Jul 31 18:45:02.290982 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.290961 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a9c6956f-b84a-41c3-a728-7c0f0895a651-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d86d85884-shqbj\" (UID: \"a9c6956f-b84a-41c3-a728-7c0f0895a651\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d86d85884-shqbj" Jul 31 18:45:02.291075 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.290995 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3bea09ae-5eb5-4b26-a6d3-793ed2defb45-registry-tls\") pod \"image-registry-b4695766f-dvcj6\" (UID: \"3bea09ae-5eb5-4b26-a6d3-793ed2defb45\") " pod="openshift-image-registry/image-registry-b4695766f-dvcj6" Jul 31 18:45:02.292423 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.292405 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" Jul 31 18:45:02.295900 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.295875 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:02.297156 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.297133 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjjzn\" (UniqueName: \"kubernetes.io/projected/a9c6956f-b84a-41c3-a728-7c0f0895a651-kube-api-access-gjjzn\") pod \"klusterlet-addon-workmgr-6d86d85884-shqbj\" (UID: \"a9c6956f-b84a-41c3-a728-7c0f0895a651\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d86d85884-shqbj" Jul 31 18:45:02.297933 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.297871 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-6fcww" Jul 31 18:45:02.299388 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.299349 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3bea09ae-5eb5-4b26-a6d3-793ed2defb45-bound-sa-token\") pod \"image-registry-b4695766f-dvcj6\" (UID: \"3bea09ae-5eb5-4b26-a6d3-793ed2defb45\") " pod="openshift-image-registry/image-registry-b4695766f-dvcj6" Jul 31 18:45:02.299540 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.299525 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws5xv\" (UniqueName: \"kubernetes.io/projected/e3d33d0d-4656-4016-be66-99d9aa6e0994-kube-api-access-ws5xv\") pod \"downloads-77c974ffcf-hb2hk\" (UID: \"e3d33d0d-4656-4016-be66-99d9aa6e0994\") " pod="openshift-console/downloads-77c974ffcf-hb2hk" Jul 31 18:45:02.299950 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.299935 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xj5s\" (UniqueName: \"kubernetes.io/projected/3bea09ae-5eb5-4b26-a6d3-793ed2defb45-kube-api-access-8xj5s\") pod \"image-registry-b4695766f-dvcj6\" (UID: \"3bea09ae-5eb5-4b26-a6d3-793ed2defb45\") " pod="openshift-image-registry/image-registry-b4695766f-dvcj6" Jul 31 18:45:02.344076 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.344049 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b6f988b74-gvnd4" Jul 31 18:45:02.350271 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.350208 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d86d85884-shqbj" Jul 31 18:45:02.388737 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.388709 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-ca-trust-extracted\") pod \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " Jul 31 18:45:02.388864 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.388764 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-bound-sa-token\") pod \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " Jul 31 18:45:02.388864 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.388807 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-installation-pull-secrets\") pod \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " Jul 31 18:45:02.388864 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.388855 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-image-registry-private-configuration\") pod \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " Jul 31 18:45:02.389011 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.388895 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-registry-certificates\") pod \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " Jul 31 18:45:02.389011 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.388954 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnsm5\" (UniqueName: \"kubernetes.io/projected/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-kube-api-access-jnsm5\") pod \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " Jul 31 18:45:02.389011 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.388998 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-trusted-ca\") pod \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\" (UID: \"cdc4eaf7-429f-4fc0-be49-1c4913cfdb85\") " Jul 31 18:45:02.389397 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.389367 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "cdc4eaf7-429f-4fc0-be49-1c4913cfdb85" (UID: "cdc4eaf7-429f-4fc0-be49-1c4913cfdb85"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 18:45:02.389666 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.389625 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "cdc4eaf7-429f-4fc0-be49-1c4913cfdb85" (UID: "cdc4eaf7-429f-4fc0-be49-1c4913cfdb85"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 18:45:02.390105 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.390070 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "cdc4eaf7-429f-4fc0-be49-1c4913cfdb85" (UID: "cdc4eaf7-429f-4fc0-be49-1c4913cfdb85"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 18:45:02.397915 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.397853 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "cdc4eaf7-429f-4fc0-be49-1c4913cfdb85" (UID: "cdc4eaf7-429f-4fc0-be49-1c4913cfdb85"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:45:02.398015 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.397932 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "cdc4eaf7-429f-4fc0-be49-1c4913cfdb85" (UID: "cdc4eaf7-429f-4fc0-be49-1c4913cfdb85"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 18:45:02.398216 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.398089 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-kube-api-access-jnsm5" (OuterVolumeSpecName: "kube-api-access-jnsm5") pod "cdc4eaf7-429f-4fc0-be49-1c4913cfdb85" (UID: "cdc4eaf7-429f-4fc0-be49-1c4913cfdb85"). InnerVolumeSpecName "kube-api-access-jnsm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 18:45:02.399022 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.398638 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:02.399022 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.398958 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "cdc4eaf7-429f-4fc0-be49-1c4913cfdb85" (UID: "cdc4eaf7-429f-4fc0-be49-1c4913cfdb85"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:45:02.401899 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.401875 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-hb2hk" Jul 31 18:45:02.412662 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.412426 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4pbfx\"" Jul 31 18:45:02.421472 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.421424 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b4695766f-dvcj6" Jul 31 18:45:02.439310 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.439274 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-6fcww"] Jul 31 18:45:02.442637 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:45:02.442597 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b029477_8e2e_46ba_8209_d3cd2f49f582.slice/crio-f33f8ee8a17ef2bb588859553526c8d6640996f2247da29930190eb2c4af1fd6 WatchSource:0}: Error finding container f33f8ee8a17ef2bb588859553526c8d6640996f2247da29930190eb2c4af1fd6: Status 404 returned error can't find the container with id f33f8ee8a17ef2bb588859553526c8d6640996f2247da29930190eb2c4af1fd6 Jul 31 18:45:02.491425 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.491289 2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-image-registry-private-configuration\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:45:02.491425 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.491323 2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-registry-certificates\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:45:02.491425 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.491339 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnsm5\" (UniqueName: \"kubernetes.io/projected/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-kube-api-access-jnsm5\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:45:02.491425 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.491353 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-trusted-ca\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:45:02.491425 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.491366 2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-ca-trust-extracted\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:45:02.491425 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.491383 2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-bound-sa-token\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:45:02.491425 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.491395 2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-installation-pull-secrets\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:45:02.497614 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.497567 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:02.503471 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.502923 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d86d85884-shqbj"] Jul 31 18:45:02.508908 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:45:02.508881 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c6956f_b84a_41c3_a728_7c0f0895a651.slice/crio-130689cf395206f47d167ebd172422549e8f9a0a952ca5365693a70ab1550978 WatchSource:0}: Error finding container 130689cf395206f47d167ebd172422549e8f9a0a952ca5365693a70ab1550978: Status 404 returned error can't find the container with id 130689cf395206f47d167ebd172422549e8f9a0a952ca5365693a70ab1550978 Jul 31 18:45:02.525407 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.525372 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b6f988b74-gvnd4"] Jul 31 18:45:02.528870 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:45:02.528844 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5efb0219_e98d_45a6_ad94_122359fc7cda.slice/crio-adcf5f16cb8c15caab8d5db08a29829cf96983a35cf1fa96c193f334078661a2 WatchSource:0}: Error finding container adcf5f16cb8c15caab8d5db08a29829cf96983a35cf1fa96c193f334078661a2: Status 404 returned error can't find the container with id adcf5f16cb8c15caab8d5db08a29829cf96983a35cf1fa96c193f334078661a2 Jul 31 18:45:02.575570 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.575539 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b4695766f-dvcj6"] Jul 31 18:45:02.578579 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:45:02.578551 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bea09ae_5eb5_4b26_a6d3_793ed2defb45.slice/crio-ad8621df9acb2364b25a17d6468bb7e473493b20f14aa69b7f36903c434e8548 WatchSource:0}: Error finding container ad8621df9acb2364b25a17d6468bb7e473493b20f14aa69b7f36903c434e8548: Status 404 returned error can't find the container with id ad8621df9acb2364b25a17d6468bb7e473493b20f14aa69b7f36903c434e8548 Jul 31 18:45:02.592267 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:02.591630 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-hb2hk"] Jul 31 18:45:03.291606 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:03.291525 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-hb2hk" event={"ID":"e3d33d0d-4656-4016-be66-99d9aa6e0994","Type":"ContainerStarted","Data":"1a33c71d85359cdab568e9a6af9caffb43ea1fc134f77407f092a1226c710565"} Jul 31 18:45:03.293389 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:03.293355 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b4695766f-dvcj6" event={"ID":"3bea09ae-5eb5-4b26-a6d3-793ed2defb45","Type":"ContainerStarted","Data":"06e3fedd8ff4355f4f7dc7d42dd9c0b9d80c71ad6ce148fd557af6d1e651c8d5"} Jul 31 18:45:03.293519 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:03.293410 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b4695766f-dvcj6" event={"ID":"3bea09ae-5eb5-4b26-a6d3-793ed2defb45","Type":"ContainerStarted","Data":"ad8621df9acb2364b25a17d6468bb7e473493b20f14aa69b7f36903c434e8548"} Jul 31 18:45:03.293519 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:03.293482 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b4695766f-dvcj6" Jul 31 18:45:03.294897 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:03.294847 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-6fcww" event={"ID":"0b029477-8e2e-46ba-8209-d3cd2f49f582","Type":"ContainerStarted","Data":"f33f8ee8a17ef2bb588859553526c8d6640996f2247da29930190eb2c4af1fd6"} Jul 31 18:45:03.296166 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:03.296141 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d86d85884-shqbj" event={"ID":"a9c6956f-b84a-41c3-a728-7c0f0895a651","Type":"ContainerStarted","Data":"130689cf395206f47d167ebd172422549e8f9a0a952ca5365693a70ab1550978"} Jul 31 18:45:03.297626 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:03.297593 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b6f988b74-gvnd4" event={"ID":"5efb0219-e98d-45a6-ad94-122359fc7cda","Type":"ContainerStarted","Data":"adcf5f16cb8c15caab8d5db08a29829cf96983a35cf1fa96c193f334078661a2"} Jul 31 18:45:03.297728 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:03.297677 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bf57fb7b8-l6h5t" Jul 31 18:45:03.315601 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:03.315555 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-b4695766f-dvcj6" podStartSLOduration=1.315540218 podStartE2EDuration="1.315540218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:03.315164953 +0000 UTC m=+79.953154754" watchObservedRunningTime="2026-07-31 18:45:03.315540218 +0000 UTC m=+79.953530015" Jul 31 18:45:03.345373 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:03.345338 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6bf57fb7b8-l6h5t"] Jul 31 18:45:03.352289 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:03.352256 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6bf57fb7b8-l6h5t"] Jul 31 18:45:03.501286 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:03.501247 2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85-registry-tls\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:45:03.885680 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:03.885605 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdc4eaf7-429f-4fc0-be49-1c4913cfdb85" path="/var/lib/kubelet/pods/cdc4eaf7-429f-4fc0-be49-1c4913cfdb85/volumes" Jul 31 18:45:04.302566 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:04.302528 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-6fcww" event={"ID":"0b029477-8e2e-46ba-8209-d3cd2f49f582","Type":"ContainerStarted","Data":"b6f4242d88f98c98949d4f2186e7cbfc02f0549838b0d17adef698be2581ab99"} Jul 31 18:45:04.320386 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:04.320329 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-f77f48c6d-6fcww" podStartSLOduration=1.839261231 podStartE2EDuration="3.320306277s" totalImagesPullingTime="1.481045046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:04.319470498 +0000 UTC m=+80.957460299" watchObservedRunningTime="2026-07-31 18:45:04.320306277 +0000 UTC m=+80.958296076" Jul 31 18:45:04.453224 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:04.453181 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:04.553463 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:04.553379 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:04.611609 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:04.611572 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/45c0e651-aba9-4741-84db-8a06cd82c46e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-m76n7\" (UID: \"45c0e651-aba9-4741-84db-8a06cd82c46e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m76n7" Jul 31 18:45:04.614157 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:04.614127 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/45c0e651-aba9-4741-84db-8a06cd82c46e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-m76n7\" (UID: \"45c0e651-aba9-4741-84db-8a06cd82c46e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m76n7" Jul 31 18:45:04.773634 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:04.773588 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8rmqt\"" Jul 31 18:45:04.782220 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:04.782196 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m76n7" Jul 31 18:45:04.913808 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:04.913771 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8c11319-084e-4318-9e92-5a4a68d8c916-service-ca-bundle\") pod \"router-default-685dfffc96-vrmgp\" (UID: \"a8c11319-084e-4318-9e92-5a4a68d8c916\") " pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:45:04.914183 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:04.913970 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8c11319-084e-4318-9e92-5a4a68d8c916-metrics-certs\") pod \"router-default-685dfffc96-vrmgp\" (UID: \"a8c11319-084e-4318-9e92-5a4a68d8c916\") " pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:45:04.914701 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:04.914649 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8c11319-084e-4318-9e92-5a4a68d8c916-service-ca-bundle\") pod \"router-default-685dfffc96-vrmgp\" (UID: \"a8c11319-084e-4318-9e92-5a4a68d8c916\") " pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:45:04.926222 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:04.926065 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8c11319-084e-4318-9e92-5a4a68d8c916-metrics-certs\") pod \"router-default-685dfffc96-vrmgp\" (UID: \"a8c11319-084e-4318-9e92-5a4a68d8c916\") " pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:45:04.974144 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:04.973660 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m76n7"] Jul 31 18:45:04.983514 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:45:04.981577 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c0e651_aba9_4741_84db_8a06cd82c46e.slice/crio-45b6987f760dbd88b85a28ff94833e14ca0bcda7bf30574376b17482ccbdcbca WatchSource:0}: Error finding container 45b6987f760dbd88b85a28ff94833e14ca0bcda7bf30574376b17482ccbdcbca: Status 404 returned error can't find the container with id 45b6987f760dbd88b85a28ff94833e14ca0bcda7bf30574376b17482ccbdcbca Jul 31 18:45:05.063330 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:05.063296 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-s6rjx\"" Jul 31 18:45:05.072402 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:05.072375 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:45:05.241053 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:05.240979 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-685dfffc96-vrmgp"] Jul 31 18:45:05.246542 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:45:05.246508 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c11319_084e_4318_9e92_5a4a68d8c916.slice/crio-fcd6e899b5992b7e91f7680a21a68ba588e8bf8579e1275058ffcce1a7f827d4 WatchSource:0}: Error finding container fcd6e899b5992b7e91f7680a21a68ba588e8bf8579e1275058ffcce1a7f827d4: Status 404 returned error can't find the container with id fcd6e899b5992b7e91f7680a21a68ba588e8bf8579e1275058ffcce1a7f827d4 Jul 31 18:45:05.309580 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:05.309550 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m76n7" event={"ID":"45c0e651-aba9-4741-84db-8a06cd82c46e","Type":"ContainerStarted","Data":"45b6987f760dbd88b85a28ff94833e14ca0bcda7bf30574376b17482ccbdcbca"} Jul 31 18:45:05.311080 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:05.311049 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-685dfffc96-vrmgp" event={"ID":"a8c11319-084e-4318-9e92-5a4a68d8c916","Type":"ContainerStarted","Data":"fcd6e899b5992b7e91f7680a21a68ba588e8bf8579e1275058ffcce1a7f827d4"} Jul 31 18:45:06.272399 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:06.272182 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dgmn6" Jul 31 18:45:06.320283 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:06.319883 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-685dfffc96-vrmgp" event={"ID":"a8c11319-084e-4318-9e92-5a4a68d8c916","Type":"ContainerStarted","Data":"4415b317e16fb83f834db8e15976a49bcfbb49becb19366ccc7c3f0c17e1ae3d"} Jul 31 18:45:06.398938 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:06.398878 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-685dfffc96-vrmgp" podStartSLOduration=34.398855842 podStartE2EDuration="34.398855842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:06.397947048 +0000 UTC m=+83.035936849" watchObservedRunningTime="2026-07-31 18:45:06.398855842 +0000 UTC m=+83.036845643" Jul 31 18:45:07.073291 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:07.073253 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:45:07.075932 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:07.075719 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:45:07.325028 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:07.324974 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:45:07.326498 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:07.326474 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-685dfffc96-vrmgp" Jul 31 18:45:07.788483 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:07.788447 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:08.340776 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:08.340745 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b6f988b74-gvnd4" event={"ID":"5efb0219-e98d-45a6-ad94-122359fc7cda","Type":"ContainerStarted","Data":"dcd26958f466ecefe173553276346f889fa7cba76c141f41f8baf578a5815b36"} Jul 31 18:45:08.360768 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:08.360182 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b6f988b74-gvnd4" podStartSLOduration=2.604939317 podStartE2EDuration="7.360163177s" totalImagesPullingTime="4.75522386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:08.358949031 +0000 UTC m=+84.996938830" watchObservedRunningTime="2026-07-31 18:45:08.360163177 +0000 UTC m=+84.998152980" Jul 31 18:45:09.861867 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:09.861834 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e664160-fd29-4f67-b32e-8ae37b2641be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vrdbp\" (UID: \"0e664160-fd29-4f67-b32e-8ae37b2641be\") " pod="openshift-insights/insights-runtime-extractor-vrdbp" Jul 31 18:45:09.864382 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:09.864357 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e664160-fd29-4f67-b32e-8ae37b2641be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vrdbp\" (UID: \"0e664160-fd29-4f67-b32e-8ae37b2641be\") " pod="openshift-insights/insights-runtime-extractor-vrdbp" Jul 31 18:45:09.873478 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:09.873459 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-v7tf4\"" Jul 31 18:45:09.879983 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:09.879959 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vrdbp" Jul 31 18:45:10.013221 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:10.013107 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:10.053328 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:10.053227 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vrdbp"] Jul 31 18:45:10.132873 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:10.132803 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:10.320668 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:10.320625 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:10.349302 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:10.348905 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d86d85884-shqbj" event={"ID":"a9c6956f-b84a-41c3-a728-7c0f0895a651","Type":"ContainerStarted","Data":"0013970db01e301d1ee2084596fa264e8559d0bd2d111feccdf2a0e3506e1737"} Jul 31 18:45:10.349302 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:10.349045 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d86d85884-shqbj" Jul 31 18:45:10.350956 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:10.350798 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vrdbp" event={"ID":"0e664160-fd29-4f67-b32e-8ae37b2641be","Type":"ContainerStarted","Data":"4d06ca3cc07e18bbf75a04487a604b65483d87170f77550b21e9eb87d83c2202"} Jul 31 18:45:10.350956 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:10.350853 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vrdbp" event={"ID":"0e664160-fd29-4f67-b32e-8ae37b2641be","Type":"ContainerStarted","Data":"e1974b18ccf8beccc4e38bc70bfdbe3da57cbb6fddf0648d2456c52b63d2c5ad"} Jul 31 18:45:10.351176 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:10.350979 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d86d85884-shqbj" Jul 31 18:45:10.352767 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:10.352740 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m76n7" event={"ID":"45c0e651-aba9-4741-84db-8a06cd82c46e","Type":"ContainerStarted","Data":"3ba941908d9eadcb949fa94c53e95eeb25eeff64438cc3e9da4139af66861e32"} Jul 31 18:45:10.392135 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:10.392080 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d86d85884-shqbj" podStartSLOduration=2.052436358 podStartE2EDuration="9.392062325s" totalImagesPullingTime="7.339625967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:10.390589868 +0000 UTC m=+87.028579669" watchObservedRunningTime="2026-07-31 18:45:10.392062325 +0000 UTC m=+87.030052125" Jul 31 18:45:10.436833 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:10.436782 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:10.659492 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:10.659415 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:10.924785 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:10.924713 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:11.359594 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:11.359551 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vrdbp" event={"ID":"0e664160-fd29-4f67-b32e-8ae37b2641be","Type":"ContainerStarted","Data":"3f3e8341bed487e5061d792c18cffb4f09ab45a1189ec46cf43b50fd56f9bf88"} Jul 31 18:45:13.369895 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:13.369853 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vrdbp" event={"ID":"0e664160-fd29-4f67-b32e-8ae37b2641be","Type":"ContainerStarted","Data":"66a93d3fa6a8f7d467c760ae9be458616c0d8ba9c80b342654ad522984eef36f"} Jul 31 18:45:13.399517 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:13.399461 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m76n7" podStartSLOduration=36.545086345 podStartE2EDuration="41.399442934s" totalImagesPullingTime="4.854356589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:44:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:10.443020574 +0000 UTC m=+87.081010376" watchObservedRunningTime="2026-07-31 18:45:13.399442934 +0000 UTC m=+90.037432735" Jul 31 18:45:13.400591 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:13.400553 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vrdbp" podStartSLOduration=33.152766954 podStartE2EDuration="35.400544172s" totalImagesPullingTime="2.247777218s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:44:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:13.398817612 +0000 UTC m=+90.036807423" watchObservedRunningTime="2026-07-31 18:45:13.400544172 +0000 UTC m=+90.038533972" Jul 31 18:45:13.457151 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:13.457117 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:13.535909 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:13.535867 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:13.701693 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:13.701585 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:13.800654 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:13.800602 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:13.915050 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:13.915014 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:14.994133 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:14.994096 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:15.075934 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:15.075896 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:15.166163 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:15.166125 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:15.241403 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:15.241357 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:15.431519 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:15.431431 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:15.493182 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:15.493147 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:15.949319 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:15.949283 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:17.581143 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:17.581114 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:17.642455 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:17.642416 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:17.910670 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:17.910582 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:18.040411 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.040366 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:18.151001 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.150883 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:18.216594 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.216517 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-474bp"] Jul 31 18:45:18.224209 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.224180 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-474bp" Jul 31 18:45:18.227429 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.227403 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 31 18:45:18.227429 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.227414 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 31 18:45:18.228032 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.228012 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lbzjz\"" Jul 31 18:45:18.231277 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.231230 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 31 18:45:18.240997 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.240979 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 31 18:45:18.301012 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.300970 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:18.331439 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.331403 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkkkw\" (UniqueName: \"kubernetes.io/projected/2315e49b-1ea3-4d46-abe6-9182a23ba0c4-kube-api-access-tkkkw\") pod \"node-exporter-474bp\" (UID: \"2315e49b-1ea3-4d46-abe6-9182a23ba0c4\") " pod="openshift-monitoring/node-exporter-474bp" Jul 31 18:45:18.331726 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.331449 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2315e49b-1ea3-4d46-abe6-9182a23ba0c4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-474bp\" (UID: \"2315e49b-1ea3-4d46-abe6-9182a23ba0c4\") " pod="openshift-monitoring/node-exporter-474bp" Jul 31 18:45:18.331726 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.331477 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2315e49b-1ea3-4d46-abe6-9182a23ba0c4-metrics-client-ca\") pod \"node-exporter-474bp\" (UID: \"2315e49b-1ea3-4d46-abe6-9182a23ba0c4\") " pod="openshift-monitoring/node-exporter-474bp" Jul 31 18:45:18.331726 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.331563 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2315e49b-1ea3-4d46-abe6-9182a23ba0c4-node-exporter-accelerators-collector-config\") pod \"node-exporter-474bp\" (UID: \"2315e49b-1ea3-4d46-abe6-9182a23ba0c4\") " pod="openshift-monitoring/node-exporter-474bp" Jul 31 18:45:18.331726 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.331612 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2315e49b-1ea3-4d46-abe6-9182a23ba0c4-node-exporter-wtmp\") pod \"node-exporter-474bp\" (UID: \"2315e49b-1ea3-4d46-abe6-9182a23ba0c4\") " pod="openshift-monitoring/node-exporter-474bp" Jul 31 18:45:18.331726 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.331650 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2315e49b-1ea3-4d46-abe6-9182a23ba0c4-node-exporter-tls\") pod \"node-exporter-474bp\" (UID: \"2315e49b-1ea3-4d46-abe6-9182a23ba0c4\") " pod="openshift-monitoring/node-exporter-474bp" Jul 31 18:45:18.331726 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.331679 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2315e49b-1ea3-4d46-abe6-9182a23ba0c4-root\") pod \"node-exporter-474bp\" (UID: \"2315e49b-1ea3-4d46-abe6-9182a23ba0c4\") " pod="openshift-monitoring/node-exporter-474bp" Jul 31 18:45:18.331726 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.331703 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2315e49b-1ea3-4d46-abe6-9182a23ba0c4-node-exporter-textfile\") pod \"node-exporter-474bp\" (UID: \"2315e49b-1ea3-4d46-abe6-9182a23ba0c4\") " pod="openshift-monitoring/node-exporter-474bp" Jul 31 18:45:18.332027 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.331782 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2315e49b-1ea3-4d46-abe6-9182a23ba0c4-sys\") pod \"node-exporter-474bp\" (UID: \"2315e49b-1ea3-4d46-abe6-9182a23ba0c4\") " pod="openshift-monitoring/node-exporter-474bp" Jul 31 18:45:18.391758 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.391717 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:18.433284 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.433228 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2315e49b-1ea3-4d46-abe6-9182a23ba0c4-node-exporter-wtmp\") pod \"node-exporter-474bp\" (UID: \"2315e49b-1ea3-4d46-abe6-9182a23ba0c4\") " pod="openshift-monitoring/node-exporter-474bp" Jul 31 18:45:18.433482 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.433309 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2315e49b-1ea3-4d46-abe6-9182a23ba0c4-node-exporter-tls\") pod \"node-exporter-474bp\" (UID: \"2315e49b-1ea3-4d46-abe6-9182a23ba0c4\") " pod="openshift-monitoring/node-exporter-474bp" Jul 31 18:45:18.433482 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.433340 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2315e49b-1ea3-4d46-abe6-9182a23ba0c4-root\") pod \"node-exporter-474bp\" (UID: \"2315e49b-1ea3-4d46-abe6-9182a23ba0c4\") " pod="openshift-monitoring/node-exporter-474bp" Jul 31 18:45:18.433482 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.433365 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2315e49b-1ea3-4d46-abe6-9182a23ba0c4-node-exporter-textfile\") pod \"node-exporter-474bp\" (UID: \"2315e49b-1ea3-4d46-abe6-9182a23ba0c4\") " pod="openshift-monitoring/node-exporter-474bp" Jul 31 18:45:18.433482 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.433392 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2315e49b-1ea3-4d46-abe6-9182a23ba0c4-node-exporter-wtmp\") pod \"node-exporter-474bp\" (UID: \"2315e49b-1ea3-4d46-abe6-9182a23ba0c4\") " pod="openshift-monitoring/node-exporter-474bp" Jul 31 18:45:18.433482 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.433399 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2315e49b-1ea3-4d46-abe6-9182a23ba0c4-sys\") pod \"node-exporter-474bp\" (UID: \"2315e49b-1ea3-4d46-abe6-9182a23ba0c4\") " pod="openshift-monitoring/node-exporter-474bp" Jul 31 18:45:18.433482 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.433439 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2315e49b-1ea3-4d46-abe6-9182a23ba0c4-sys\") pod \"node-exporter-474bp\" (UID: \"2315e49b-1ea3-4d46-abe6-9182a23ba0c4\") " pod="openshift-monitoring/node-exporter-474bp" Jul 31 18:45:18.433482 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.433456 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2315e49b-1ea3-4d46-abe6-9182a23ba0c4-root\") pod \"node-exporter-474bp\" (UID: \"2315e49b-1ea3-4d46-abe6-9182a23ba0c4\") " pod="openshift-monitoring/node-exporter-474bp" Jul 31 18:45:18.433482 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.433465 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkkkw\" (UniqueName: \"kubernetes.io/projected/2315e49b-1ea3-4d46-abe6-9182a23ba0c4-kube-api-access-tkkkw\") pod \"node-exporter-474bp\" (UID: \"2315e49b-1ea3-4d46-abe6-9182a23ba0c4\") " pod="openshift-monitoring/node-exporter-474bp" Jul 31 18:45:18.433888 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.433516 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2315e49b-1ea3-4d46-abe6-9182a23ba0c4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-474bp\" (UID: \"2315e49b-1ea3-4d46-abe6-9182a23ba0c4\") " pod="openshift-monitoring/node-exporter-474bp" Jul 31 18:45:18.433888 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.433545 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2315e49b-1ea3-4d46-abe6-9182a23ba0c4-metrics-client-ca\") pod \"node-exporter-474bp\" (UID: \"2315e49b-1ea3-4d46-abe6-9182a23ba0c4\") " pod="openshift-monitoring/node-exporter-474bp" Jul 31 18:45:18.433888 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.433618 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2315e49b-1ea3-4d46-abe6-9182a23ba0c4-node-exporter-accelerators-collector-config\") pod \"node-exporter-474bp\" (UID: \"2315e49b-1ea3-4d46-abe6-9182a23ba0c4\") " pod="openshift-monitoring/node-exporter-474bp" Jul 31 18:45:18.433888 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.433776 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2315e49b-1ea3-4d46-abe6-9182a23ba0c4-node-exporter-textfile\") pod \"node-exporter-474bp\" (UID: \"2315e49b-1ea3-4d46-abe6-9182a23ba0c4\") " pod="openshift-monitoring/node-exporter-474bp" Jul 31 18:45:18.434285 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.434259 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2315e49b-1ea3-4d46-abe6-9182a23ba0c4-node-exporter-accelerators-collector-config\") pod \"node-exporter-474bp\" (UID: \"2315e49b-1ea3-4d46-abe6-9182a23ba0c4\") " pod="openshift-monitoring/node-exporter-474bp" Jul 31 18:45:18.434392 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.434369 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2315e49b-1ea3-4d46-abe6-9182a23ba0c4-metrics-client-ca\") pod \"node-exporter-474bp\" (UID: \"2315e49b-1ea3-4d46-abe6-9182a23ba0c4\") " pod="openshift-monitoring/node-exporter-474bp" Jul 31 18:45:18.436142 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.436118 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2315e49b-1ea3-4d46-abe6-9182a23ba0c4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-474bp\" (UID: \"2315e49b-1ea3-4d46-abe6-9182a23ba0c4\") " pod="openshift-monitoring/node-exporter-474bp" Jul 31 18:45:18.436414 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.436388 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2315e49b-1ea3-4d46-abe6-9182a23ba0c4-node-exporter-tls\") pod \"node-exporter-474bp\" (UID: \"2315e49b-1ea3-4d46-abe6-9182a23ba0c4\") " pod="openshift-monitoring/node-exporter-474bp" Jul 31 18:45:18.443898 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.443872 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkkkw\" (UniqueName: \"kubernetes.io/projected/2315e49b-1ea3-4d46-abe6-9182a23ba0c4-kube-api-access-tkkkw\") pod \"node-exporter-474bp\" (UID: \"2315e49b-1ea3-4d46-abe6-9182a23ba0c4\") " pod="openshift-monitoring/node-exporter-474bp" Jul 31 18:45:18.491569 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.491459 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:18.534842 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.534809 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-474bp" Jul 31 18:45:18.595856 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.595815 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:18.794687 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:18.794646 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:19.077219 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:19.077142 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:19.222810 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:19.222752 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:19.385656 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:19.385562 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:19.500574 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:19.500533 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:20.414083 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:20.414048 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:20.430317 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:45:20.430283 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2315e49b_1ea3_4d46_abe6_9182a23ba0c4.slice/crio-4568210fd9c78eecf20c42d0fb705febd53559c80d10b9b847f0413bfe3fe652 WatchSource:0}: Error finding container 4568210fd9c78eecf20c42d0fb705febd53559c80d10b9b847f0413bfe3fe652: Status 404 returned error can't find the container with id 4568210fd9c78eecf20c42d0fb705febd53559c80d10b9b847f0413bfe3fe652 Jul 31 18:45:20.514632 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:20.514576 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:21.142950 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:21.142191 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:21.227983 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:21.227931 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:21.396802 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:21.396657 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:21.404217 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:21.404139 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-474bp" event={"ID":"2315e49b-1ea3-4d46-abe6-9182a23ba0c4","Type":"ContainerStarted","Data":"4568210fd9c78eecf20c42d0fb705febd53559c80d10b9b847f0413bfe3fe652"} Jul 31 18:45:21.408045 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:21.406341 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-hb2hk" event={"ID":"e3d33d0d-4656-4016-be66-99d9aa6e0994","Type":"ContainerStarted","Data":"3c510fb26482b5fd61c87caf684062674b2e051b4c1e647dee39ce8935494bc1"} Jul 31 18:45:21.408348 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:21.408325 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-77c974ffcf-hb2hk" Jul 31 18:45:21.424552 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:21.424496 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-77c974ffcf-hb2hk" podStartSLOduration=1.215633755 podStartE2EDuration="19.424478393s" totalImagesPullingTime="18.208844638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:45:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:21.423648097 +0000 UTC m=+98.061637899" watchObservedRunningTime="2026-07-31 18:45:21.424478393 +0000 UTC m=+98.062468195" Jul 31 18:45:21.427273 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:21.427233 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-77c974ffcf-hb2hk" Jul 31 18:45:21.635075 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:21.635037 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:22.412207 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:22.412114 2565 generic.go:356] "Generic (PLEG): container finished" podID="2315e49b-1ea3-4d46-abe6-9182a23ba0c4" containerID="d308429d7644b1248e0540620249619a2a71a8b2b9bd15bfed9568efcdd7cc96" exitCode=0 Jul 31 18:45:22.412401 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:22.412202 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-474bp" event={"ID":"2315e49b-1ea3-4d46-abe6-9182a23ba0c4","Type":"ContainerDied","Data":"d308429d7644b1248e0540620249619a2a71a8b2b9bd15bfed9568efcdd7cc96"} Jul 31 18:45:22.908278 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:22.908229 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:23.008192 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:23.008158 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:23.116446 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:23.116400 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:23.419875 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:23.419796 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-474bp" event={"ID":"2315e49b-1ea3-4d46-abe6-9182a23ba0c4","Type":"ContainerStarted","Data":"3d945d1fd0df97fca049290fd3946ec76fee4e9395d0a181abdf435cdb924fcb"} Jul 31 18:45:23.419875 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:23.419853 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-474bp" event={"ID":"2315e49b-1ea3-4d46-abe6-9182a23ba0c4","Type":"ContainerStarted","Data":"2a4df4f70dc5af2409128b841ed4fa9ac1eb38f9c8bb2f32a896e350e14f69b9"} Jul 31 18:45:23.450842 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:23.450781 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-474bp" podStartSLOduration=3.928718285 podStartE2EDuration="5.450747926s" totalImagesPullingTime="1.411344549s" totalInitContainerRuntime="110.685092ms" isStatefulPod=true podCreationTimestamp="2026-07-31 18:45:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:23.447719755 +0000 UTC m=+100.085709584" watchObservedRunningTime="2026-07-31 18:45:23.450747926 +0000 UTC m=+100.088737726" Jul 31 18:45:24.307575 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:24.307544 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b4695766f-dvcj6" Jul 31 18:45:24.352752 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:24.352712 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:24.473349 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:24.473306 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:24.585902 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:24.585818 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:24.703422 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:24.703385 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:24.984449 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:24.984353 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:25.175431 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:25.175394 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:25.528390 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:25.528313 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:25.618613 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:25.618577 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:27.272589 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:27.272556 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xqlm2" Jul 31 18:45:29.605900 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:29.605865 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:29.705780 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:29.705721 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:30.331057 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.331017 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:30.391915 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.391881 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:30.641277 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.641180 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-867bb78f55-dq6bh"] Jul 31 18:45:30.673745 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.673717 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-867bb78f55-dq6bh"] Jul 31 18:45:30.673884 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.673822 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-867bb78f55-dq6bh" Jul 31 18:45:30.675855 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.675834 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 31 18:45:30.675988 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.675897 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 31 18:45:30.675988 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.675906 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 31 18:45:30.676183 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.676167 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 31 18:45:30.676440 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.676341 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 31 18:45:30.676530 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.676488 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kzdcj\"" Jul 31 18:45:30.684060 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.684031 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 31 18:45:30.747100 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.747070 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ebd8658-928b-486a-a597-cb258484c6b1-console-serving-cert\") pod \"console-867bb78f55-dq6bh\" (UID: \"9ebd8658-928b-486a-a597-cb258484c6b1\") " pod="openshift-console/console-867bb78f55-dq6bh" Jul 31 18:45:30.747219 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.747109 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9ebd8658-928b-486a-a597-cb258484c6b1-oauth-serving-cert\") pod \"console-867bb78f55-dq6bh\" (UID: \"9ebd8658-928b-486a-a597-cb258484c6b1\") " pod="openshift-console/console-867bb78f55-dq6bh" Jul 31 18:45:30.747219 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.747157 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9ebd8658-928b-486a-a597-cb258484c6b1-console-oauth-config\") pod \"console-867bb78f55-dq6bh\" (UID: \"9ebd8658-928b-486a-a597-cb258484c6b1\") " pod="openshift-console/console-867bb78f55-dq6bh" Jul 31 18:45:30.747219 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.747176 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhbfl\" (UniqueName: \"kubernetes.io/projected/9ebd8658-928b-486a-a597-cb258484c6b1-kube-api-access-jhbfl\") pod \"console-867bb78f55-dq6bh\" (UID: \"9ebd8658-928b-486a-a597-cb258484c6b1\") " pod="openshift-console/console-867bb78f55-dq6bh" Jul 31 18:45:30.747219 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.747207 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9ebd8658-928b-486a-a597-cb258484c6b1-console-config\") pod \"console-867bb78f55-dq6bh\" (UID: \"9ebd8658-928b-486a-a597-cb258484c6b1\") " pod="openshift-console/console-867bb78f55-dq6bh" Jul 31 18:45:30.747409 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.747264 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9ebd8658-928b-486a-a597-cb258484c6b1-service-ca\") pod \"console-867bb78f55-dq6bh\" (UID: \"9ebd8658-928b-486a-a597-cb258484c6b1\") " pod="openshift-console/console-867bb78f55-dq6bh" Jul 31 18:45:30.747409 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.747289 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ebd8658-928b-486a-a597-cb258484c6b1-trusted-ca-bundle\") pod \"console-867bb78f55-dq6bh\" (UID: \"9ebd8658-928b-486a-a597-cb258484c6b1\") " pod="openshift-console/console-867bb78f55-dq6bh" Jul 31 18:45:30.847963 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.847929 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9ebd8658-928b-486a-a597-cb258484c6b1-console-config\") pod \"console-867bb78f55-dq6bh\" (UID: \"9ebd8658-928b-486a-a597-cb258484c6b1\") " pod="openshift-console/console-867bb78f55-dq6bh" Jul 31 18:45:30.847963 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.847961 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9ebd8658-928b-486a-a597-cb258484c6b1-service-ca\") pod \"console-867bb78f55-dq6bh\" (UID: \"9ebd8658-928b-486a-a597-cb258484c6b1\") " pod="openshift-console/console-867bb78f55-dq6bh" Jul 31 18:45:30.848162 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.847977 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ebd8658-928b-486a-a597-cb258484c6b1-trusted-ca-bundle\") pod \"console-867bb78f55-dq6bh\" (UID: \"9ebd8658-928b-486a-a597-cb258484c6b1\") " pod="openshift-console/console-867bb78f55-dq6bh" Jul 31 18:45:30.848162 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.848009 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ebd8658-928b-486a-a597-cb258484c6b1-console-serving-cert\") pod \"console-867bb78f55-dq6bh\" (UID: \"9ebd8658-928b-486a-a597-cb258484c6b1\") " pod="openshift-console/console-867bb78f55-dq6bh" Jul 31 18:45:30.848162 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.848037 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9ebd8658-928b-486a-a597-cb258484c6b1-oauth-serving-cert\") pod \"console-867bb78f55-dq6bh\" (UID: \"9ebd8658-928b-486a-a597-cb258484c6b1\") " pod="openshift-console/console-867bb78f55-dq6bh" Jul 31 18:45:30.848162 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.848067 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9ebd8658-928b-486a-a597-cb258484c6b1-console-oauth-config\") pod \"console-867bb78f55-dq6bh\" (UID: \"9ebd8658-928b-486a-a597-cb258484c6b1\") " pod="openshift-console/console-867bb78f55-dq6bh" Jul 31 18:45:30.848162 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.848085 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhbfl\" (UniqueName: \"kubernetes.io/projected/9ebd8658-928b-486a-a597-cb258484c6b1-kube-api-access-jhbfl\") pod \"console-867bb78f55-dq6bh\" (UID: \"9ebd8658-928b-486a-a597-cb258484c6b1\") " pod="openshift-console/console-867bb78f55-dq6bh" Jul 31 18:45:30.848766 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.848741 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9ebd8658-928b-486a-a597-cb258484c6b1-console-config\") pod \"console-867bb78f55-dq6bh\" (UID: \"9ebd8658-928b-486a-a597-cb258484c6b1\") " pod="openshift-console/console-867bb78f55-dq6bh" Jul 31 18:45:30.848898 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.848874 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9ebd8658-928b-486a-a597-cb258484c6b1-service-ca\") pod \"console-867bb78f55-dq6bh\" (UID: \"9ebd8658-928b-486a-a597-cb258484c6b1\") " pod="openshift-console/console-867bb78f55-dq6bh" Jul 31 18:45:30.848939 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.848919 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9ebd8658-928b-486a-a597-cb258484c6b1-oauth-serving-cert\") pod \"console-867bb78f55-dq6bh\" (UID: \"9ebd8658-928b-486a-a597-cb258484c6b1\") " pod="openshift-console/console-867bb78f55-dq6bh" Jul 31 18:45:30.848978 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.848956 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ebd8658-928b-486a-a597-cb258484c6b1-trusted-ca-bundle\") pod \"console-867bb78f55-dq6bh\" (UID: \"9ebd8658-928b-486a-a597-cb258484c6b1\") " pod="openshift-console/console-867bb78f55-dq6bh" Jul 31 18:45:30.850499 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.850477 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ebd8658-928b-486a-a597-cb258484c6b1-console-serving-cert\") pod \"console-867bb78f55-dq6bh\" (UID: \"9ebd8658-928b-486a-a597-cb258484c6b1\") " pod="openshift-console/console-867bb78f55-dq6bh" Jul 31 18:45:30.850568 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.850500 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9ebd8658-928b-486a-a597-cb258484c6b1-console-oauth-config\") pod \"console-867bb78f55-dq6bh\" (UID: \"9ebd8658-928b-486a-a597-cb258484c6b1\") " pod="openshift-console/console-867bb78f55-dq6bh" Jul 31 18:45:30.856095 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.856074 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhbfl\" (UniqueName: \"kubernetes.io/projected/9ebd8658-928b-486a-a597-cb258484c6b1-kube-api-access-jhbfl\") pod \"console-867bb78f55-dq6bh\" (UID: \"9ebd8658-928b-486a-a597-cb258484c6b1\") " pod="openshift-console/console-867bb78f55-dq6bh" Jul 31 18:45:30.986826 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:30.986741 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-867bb78f55-dq6bh" Jul 31 18:45:31.150044 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:31.150009 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-867bb78f55-dq6bh"] Jul 31 18:45:31.153612 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:45:31.153588 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ebd8658_928b_486a_a597_cb258484c6b1.slice/crio-07d710750f7df10a60ffe4943e868aadf20721d41df9c8e6b26389267f64f5dc WatchSource:0}: Error finding container 07d710750f7df10a60ffe4943e868aadf20721d41df9c8e6b26389267f64f5dc: Status 404 returned error can't find the container with id 07d710750f7df10a60ffe4943e868aadf20721d41df9c8e6b26389267f64f5dc Jul 31 18:45:31.430656 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:31.430624 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:31.452298 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:31.452265 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-867bb78f55-dq6bh" event={"ID":"9ebd8658-928b-486a-a597-cb258484c6b1","Type":"ContainerStarted","Data":"07d710750f7df10a60ffe4943e868aadf20721d41df9c8e6b26389267f64f5dc"} Jul 31 18:45:31.527642 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:31.527602 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:34.464580 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:34.464548 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-867bb78f55-dq6bh" event={"ID":"9ebd8658-928b-486a-a597-cb258484c6b1","Type":"ContainerStarted","Data":"e4bd3cdb568b97d3b57ce25c4aa423a776e4c78d33456ef31028dd444224c131"} Jul 31 18:45:34.484055 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:34.484001 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-867bb78f55-dq6bh" podStartSLOduration=1.367669702 podStartE2EDuration="4.483985218s" totalImagesPullingTime="3.116315516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:45:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:34.48309379 +0000 UTC m=+111.121083590" watchObservedRunningTime="2026-07-31 18:45:34.483985218 +0000 UTC m=+111.121975017" Jul 31 18:45:37.248839 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:37.248801 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:37.342726 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:37.342666 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:39.604710 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:39.604667 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:39.689200 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:39.689157 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:40.986996 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:40.986964 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-867bb78f55-dq6bh" Jul 31 18:45:40.987406 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:40.987013 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-867bb78f55-dq6bh" Jul 31 18:45:40.991732 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:40.991709 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-867bb78f55-dq6bh" Jul 31 18:45:41.492185 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:41.492155 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-867bb78f55-dq6bh" Jul 31 18:45:41.803736 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:41.803703 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:41.952165 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:41.952122 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:43.496526 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:43.496493 2565 generic.go:356] "Generic (PLEG): container finished" podID="80e51b4d-e2a8-4f98-a568-1c21551ed35c" containerID="dfbd6125afe7dd3a77273523884317ec513d1753f95834df19bfbaca1046fae5" exitCode=0 Jul 31 18:45:43.496956 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:43.496552 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6txtg" event={"ID":"80e51b4d-e2a8-4f98-a568-1c21551ed35c","Type":"ContainerDied","Data":"dfbd6125afe7dd3a77273523884317ec513d1753f95834df19bfbaca1046fae5"} Jul 31 18:45:43.496956 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:43.496866 2565 scope.go:117] "RemoveContainer" containerID="dfbd6125afe7dd3a77273523884317ec513d1753f95834df19bfbaca1046fae5" Jul 31 18:45:43.792397 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:43.792316 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:43.861673 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:43.861637 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:44.507190 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:44.507154 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-6txtg" event={"ID":"80e51b4d-e2a8-4f98-a568-1c21551ed35c","Type":"ContainerStarted","Data":"4c3debe95f1e3bc33181859452a0b6c92cd0b000e95cf5126f2010f8ecb45662"} Jul 31 18:45:46.422195 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:46.422163 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:46.493899 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:46.493866 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:53.635442 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:53.635409 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee8aa68f-72ff-4118-ad38-4da03b305feb-metrics-certs\") pod \"network-metrics-daemon-h8n5q\" (UID: \"ee8aa68f-72ff-4118-ad38-4da03b305feb\") " pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:45:53.637699 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:53.637679 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee8aa68f-72ff-4118-ad38-4da03b305feb-metrics-certs\") pod \"network-metrics-daemon-h8n5q\" (UID: \"ee8aa68f-72ff-4118-ad38-4da03b305feb\") " pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:45:53.700479 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:53.700449 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s4mbt\"" Jul 31 18:45:53.709213 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:53.709196 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8n5q" Jul 31 18:45:53.867974 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:53.867892 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h8n5q"] Jul 31 18:45:53.870197 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:45:53.870168 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee8aa68f_72ff_4118_ad38_4da03b305feb.slice/crio-2bdc943434c9b7d604a0c3ee3b306d1690407cce24c0fdfd42821cce7f23442e WatchSource:0}: Error finding container 2bdc943434c9b7d604a0c3ee3b306d1690407cce24c0fdfd42821cce7f23442e: Status 404 returned error can't find the container with id 2bdc943434c9b7d604a0c3ee3b306d1690407cce24c0fdfd42821cce7f23442e Jul 31 18:45:54.545776 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:54.545737 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h8n5q" event={"ID":"ee8aa68f-72ff-4118-ad38-4da03b305feb","Type":"ContainerStarted","Data":"2bdc943434c9b7d604a0c3ee3b306d1690407cce24c0fdfd42821cce7f23442e"} Jul 31 18:45:54.881970 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:54.881883 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:54.949434 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:54.949389 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:55.551671 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:55.551639 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h8n5q" event={"ID":"ee8aa68f-72ff-4118-ad38-4da03b305feb","Type":"ContainerStarted","Data":"eb2e89ae9c468d0533a46736ec9f8c2de55b568864ec0caaf0f7845c51433349"} Jul 31 18:45:56.558390 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:56.558353 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h8n5q" event={"ID":"ee8aa68f-72ff-4118-ad38-4da03b305feb","Type":"ContainerStarted","Data":"db6a6775d9dfd920d11e4d5b61e1c53db11b2ffada3a5036dc63a032a215b617"} Jul 31 18:45:56.577123 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:56.577066 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-h8n5q" podStartSLOduration=131.083207806 podStartE2EDuration="2m12.577049975s" totalImagesPullingTime="1.493842169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:56.575566455 +0000 UTC m=+133.213556267" watchObservedRunningTime="2026-07-31 18:45:56.577049975 +0000 UTC m=+133.215039775" Jul 31 18:45:56.764226 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:56.764186 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:45:56.835817 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:45:56.835735 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:46:01.427193 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:01.427150 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:46:01.557341 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:01.557303 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:46:06.439197 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:06.439148 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:46:07.598462 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:07.598373 2565 generic.go:356] "Generic (PLEG): container finished" podID="24f12167-7a95-45d7-bc34-9cafeec2201e" containerID="9f0aa2dfc2225c3b46f0b3fc2585f12b4c2eb61bd152a65939d07ae7cf140548" exitCode=0 Jul 31 18:46:07.598904 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:07.598451 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-rkrsg" event={"ID":"24f12167-7a95-45d7-bc34-9cafeec2201e","Type":"ContainerDied","Data":"9f0aa2dfc2225c3b46f0b3fc2585f12b4c2eb61bd152a65939d07ae7cf140548"} Jul 31 18:46:07.598904 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:07.598867 2565 scope.go:117] "RemoveContainer" containerID="9f0aa2dfc2225c3b46f0b3fc2585f12b4c2eb61bd152a65939d07ae7cf140548" Jul 31 18:46:08.604601 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:08.604563 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-rkrsg" event={"ID":"24f12167-7a95-45d7-bc34-9cafeec2201e","Type":"ContainerStarted","Data":"c080e1a7aba4a9d02ed44566198afbfbe58f427f4991cf5199caede0fdd3a524"} Jul 31 18:46:34.385606 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:34.385566 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:46:34.455280 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:34.455217 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:46:34.935214 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:34.935185 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5648497788-ctvkm"] Jul 31 18:46:34.937841 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:34.937825 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5648497788-ctvkm" Jul 31 18:46:34.948623 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:34.948597 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5648497788-ctvkm"] Jul 31 18:46:34.991803 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:34.991775 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:46:35.051944 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:35.051910 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-console-oauth-config\") pod \"console-5648497788-ctvkm\" (UID: \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\") " pod="openshift-console/console-5648497788-ctvkm" Jul 31 18:46:35.051944 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:35.051946 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-oauth-serving-cert\") pod \"console-5648497788-ctvkm\" (UID: \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\") " pod="openshift-console/console-5648497788-ctvkm" Jul 31 18:46:35.052137 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:35.051967 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-service-ca\") pod \"console-5648497788-ctvkm\" (UID: \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\") " pod="openshift-console/console-5648497788-ctvkm" Jul 31 18:46:35.052137 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:35.051986 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-console-serving-cert\") pod \"console-5648497788-ctvkm\" (UID: \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\") " pod="openshift-console/console-5648497788-ctvkm" Jul 31 18:46:35.052137 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:35.052053 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-console-config\") pod \"console-5648497788-ctvkm\" (UID: \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\") " pod="openshift-console/console-5648497788-ctvkm" Jul 31 18:46:35.052137 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:35.052084 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcvtp\" (UniqueName: \"kubernetes.io/projected/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-kube-api-access-lcvtp\") pod \"console-5648497788-ctvkm\" (UID: \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\") " pod="openshift-console/console-5648497788-ctvkm" Jul 31 18:46:35.052137 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:35.052122 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-trusted-ca-bundle\") pod \"console-5648497788-ctvkm\" (UID: \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\") " pod="openshift-console/console-5648497788-ctvkm" Jul 31 18:46:35.055668 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:35.055617 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:46:35.152828 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:35.152792 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-service-ca\") pod \"console-5648497788-ctvkm\" (UID: \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\") " pod="openshift-console/console-5648497788-ctvkm" Jul 31 18:46:35.152828 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:35.152833 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-console-serving-cert\") pod \"console-5648497788-ctvkm\" (UID: \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\") " pod="openshift-console/console-5648497788-ctvkm" Jul 31 18:46:35.153062 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:35.152854 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-console-config\") pod \"console-5648497788-ctvkm\" (UID: \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\") " pod="openshift-console/console-5648497788-ctvkm" Jul 31 18:46:35.153062 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:35.152872 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcvtp\" (UniqueName: \"kubernetes.io/projected/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-kube-api-access-lcvtp\") pod \"console-5648497788-ctvkm\" (UID: \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\") " pod="openshift-console/console-5648497788-ctvkm" Jul 31 18:46:35.153062 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:35.152929 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-trusted-ca-bundle\") pod \"console-5648497788-ctvkm\" (UID: \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\") " pod="openshift-console/console-5648497788-ctvkm" Jul 31 18:46:35.153062 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:35.153038 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-console-oauth-config\") pod \"console-5648497788-ctvkm\" (UID: \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\") " pod="openshift-console/console-5648497788-ctvkm" Jul 31 18:46:35.153288 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:35.153070 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-oauth-serving-cert\") pod \"console-5648497788-ctvkm\" (UID: \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\") " pod="openshift-console/console-5648497788-ctvkm" Jul 31 18:46:35.153663 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:35.153636 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-service-ca\") pod \"console-5648497788-ctvkm\" (UID: \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\") " pod="openshift-console/console-5648497788-ctvkm" Jul 31 18:46:35.153663 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:35.153652 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-console-config\") pod \"console-5648497788-ctvkm\" (UID: \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\") " pod="openshift-console/console-5648497788-ctvkm" Jul 31 18:46:35.153855 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:35.153702 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-oauth-serving-cert\") pod \"console-5648497788-ctvkm\" (UID: \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\") " pod="openshift-console/console-5648497788-ctvkm" Jul 31 18:46:35.153963 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:35.153938 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-trusted-ca-bundle\") pod \"console-5648497788-ctvkm\" (UID: \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\") " pod="openshift-console/console-5648497788-ctvkm" Jul 31 18:46:35.155474 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:35.155446 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-console-oauth-config\") pod \"console-5648497788-ctvkm\" (UID: \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\") " pod="openshift-console/console-5648497788-ctvkm" Jul 31 18:46:35.155576 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:35.155483 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-console-serving-cert\") pod \"console-5648497788-ctvkm\" (UID: \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\") " pod="openshift-console/console-5648497788-ctvkm" Jul 31 18:46:35.162288 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:35.162262 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcvtp\" (UniqueName: \"kubernetes.io/projected/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-kube-api-access-lcvtp\") pod \"console-5648497788-ctvkm\" (UID: \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\") " pod="openshift-console/console-5648497788-ctvkm" Jul 31 18:46:35.246329 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:35.246297 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5648497788-ctvkm" Jul 31 18:46:35.395579 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:35.395553 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5648497788-ctvkm"] Jul 31 18:46:35.398426 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:46:35.398393 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a296f33_4197_44b6_8a5e_a1ae0c83dfe8.slice/crio-2d804bce73a727f93297965a0e46346a8b68e490d4e38589059371acfd9ff823 WatchSource:0}: Error finding container 2d804bce73a727f93297965a0e46346a8b68e490d4e38589059371acfd9ff823: Status 404 returned error can't find the container with id 2d804bce73a727f93297965a0e46346a8b68e490d4e38589059371acfd9ff823 Jul 31 18:46:35.698590 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:35.698504 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5648497788-ctvkm" event={"ID":"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8","Type":"ContainerStarted","Data":"b977e9da66d9ecc8e18dcdd13d25a532ba28fced6098c5bacd1edc9f6ba083b7"} Jul 31 18:46:35.698590 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:35.698583 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5648497788-ctvkm" event={"ID":"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8","Type":"ContainerStarted","Data":"2d804bce73a727f93297965a0e46346a8b68e490d4e38589059371acfd9ff823"} Jul 31 18:46:35.722906 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:35.722850 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5648497788-ctvkm" podStartSLOduration=1.722835185 podStartE2EDuration="1.722835185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:46:35.722349308 +0000 UTC m=+172.360339107" watchObservedRunningTime="2026-07-31 18:46:35.722835185 +0000 UTC m=+172.360824985" Jul 31 18:46:45.246966 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:45.246876 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5648497788-ctvkm" Jul 31 18:46:45.246966 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:45.246950 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5648497788-ctvkm" Jul 31 18:46:45.251746 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:45.251722 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5648497788-ctvkm" Jul 31 18:46:45.739284 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:45.739190 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5648497788-ctvkm" Jul 31 18:46:45.803032 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:45.802999 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-867bb78f55-dq6bh"] Jul 31 18:46:45.979508 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:45.979473 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:46:46.084456 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:46.084414 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:46:46.180361 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:46.180312 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:46:46.289695 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:46.289660 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:46:46.359479 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:46:46.359393 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:47:01.422465 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:01.422429 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:47:01.523224 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:01.523183 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:47:01.583285 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:01.583230 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:47:10.821872 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:10.821810 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-867bb78f55-dq6bh" podUID="9ebd8658-928b-486a-a597-cb258484c6b1" containerName="console" containerID="cri-o://e4bd3cdb568b97d3b57ce25c4aa423a776e4c78d33456ef31028dd444224c131" gracePeriod=15 Jul 31 18:47:11.059756 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:11.059732 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-867bb78f55-dq6bh" Jul 31 18:47:11.248367 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:11.248341 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9ebd8658-928b-486a-a597-cb258484c6b1-console-oauth-config\") pod \"9ebd8658-928b-486a-a597-cb258484c6b1\" (UID: \"9ebd8658-928b-486a-a597-cb258484c6b1\") " Jul 31 18:47:11.248508 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:11.248388 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ebd8658-928b-486a-a597-cb258484c6b1-console-serving-cert\") pod \"9ebd8658-928b-486a-a597-cb258484c6b1\" (UID: \"9ebd8658-928b-486a-a597-cb258484c6b1\") " Jul 31 18:47:11.248508 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:11.248427 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9ebd8658-928b-486a-a597-cb258484c6b1-console-config\") pod \"9ebd8658-928b-486a-a597-cb258484c6b1\" (UID: \"9ebd8658-928b-486a-a597-cb258484c6b1\") " Jul 31 18:47:11.248508 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:11.248460 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9ebd8658-928b-486a-a597-cb258484c6b1-service-ca\") pod \"9ebd8658-928b-486a-a597-cb258484c6b1\" (UID: \"9ebd8658-928b-486a-a597-cb258484c6b1\") " Jul 31 18:47:11.248508 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:11.248500 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9ebd8658-928b-486a-a597-cb258484c6b1-oauth-serving-cert\") pod \"9ebd8658-928b-486a-a597-cb258484c6b1\" (UID: \"9ebd8658-928b-486a-a597-cb258484c6b1\") " Jul 31 18:47:11.248706 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:11.248551 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhbfl\" (UniqueName: \"kubernetes.io/projected/9ebd8658-928b-486a-a597-cb258484c6b1-kube-api-access-jhbfl\") pod \"9ebd8658-928b-486a-a597-cb258484c6b1\" (UID: \"9ebd8658-928b-486a-a597-cb258484c6b1\") " Jul 31 18:47:11.248706 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:11.248585 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ebd8658-928b-486a-a597-cb258484c6b1-trusted-ca-bundle\") pod \"9ebd8658-928b-486a-a597-cb258484c6b1\" (UID: \"9ebd8658-928b-486a-a597-cb258484c6b1\") " Jul 31 18:47:11.248922 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:11.248888 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ebd8658-928b-486a-a597-cb258484c6b1-service-ca" (OuterVolumeSpecName: "service-ca") pod "9ebd8658-928b-486a-a597-cb258484c6b1" (UID: "9ebd8658-928b-486a-a597-cb258484c6b1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 18:47:11.248990 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:11.248908 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ebd8658-928b-486a-a597-cb258484c6b1-console-config" (OuterVolumeSpecName: "console-config") pod "9ebd8658-928b-486a-a597-cb258484c6b1" (UID: "9ebd8658-928b-486a-a597-cb258484c6b1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 18:47:11.248990 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:11.248927 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ebd8658-928b-486a-a597-cb258484c6b1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9ebd8658-928b-486a-a597-cb258484c6b1" (UID: "9ebd8658-928b-486a-a597-cb258484c6b1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 18:47:11.249103 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:11.249017 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ebd8658-928b-486a-a597-cb258484c6b1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9ebd8658-928b-486a-a597-cb258484c6b1" (UID: "9ebd8658-928b-486a-a597-cb258484c6b1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 18:47:11.250657 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:11.250623 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ebd8658-928b-486a-a597-cb258484c6b1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9ebd8658-928b-486a-a597-cb258484c6b1" (UID: "9ebd8658-928b-486a-a597-cb258484c6b1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:47:11.250757 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:11.250685 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ebd8658-928b-486a-a597-cb258484c6b1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9ebd8658-928b-486a-a597-cb258484c6b1" (UID: "9ebd8658-928b-486a-a597-cb258484c6b1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:47:11.250757 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:11.250732 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ebd8658-928b-486a-a597-cb258484c6b1-kube-api-access-jhbfl" (OuterVolumeSpecName: "kube-api-access-jhbfl") pod "9ebd8658-928b-486a-a597-cb258484c6b1" (UID: "9ebd8658-928b-486a-a597-cb258484c6b1"). InnerVolumeSpecName "kube-api-access-jhbfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 18:47:11.349312 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:11.349282 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhbfl\" (UniqueName: \"kubernetes.io/projected/9ebd8658-928b-486a-a597-cb258484c6b1-kube-api-access-jhbfl\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:47:11.349312 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:11.349306 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ebd8658-928b-486a-a597-cb258484c6b1-trusted-ca-bundle\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:47:11.349312 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:11.349315 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9ebd8658-928b-486a-a597-cb258484c6b1-console-oauth-config\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:47:11.349488 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:11.349327 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ebd8658-928b-486a-a597-cb258484c6b1-console-serving-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:47:11.349488 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:11.349340 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9ebd8658-928b-486a-a597-cb258484c6b1-console-config\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:47:11.349488 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:11.349349 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9ebd8658-928b-486a-a597-cb258484c6b1-service-ca\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:47:11.349488 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:11.349358 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9ebd8658-928b-486a-a597-cb258484c6b1-oauth-serving-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:47:11.827599 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:11.827565 2565 generic.go:356] "Generic (PLEG): container finished" podID="9ebd8658-928b-486a-a597-cb258484c6b1" containerID="e4bd3cdb568b97d3b57ce25c4aa423a776e4c78d33456ef31028dd444224c131" exitCode=0 Jul 31 18:47:11.828027 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:11.827607 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-867bb78f55-dq6bh" Jul 31 18:47:11.828027 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:11.827633 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-867bb78f55-dq6bh" event={"ID":"9ebd8658-928b-486a-a597-cb258484c6b1","Type":"ContainerDied","Data":"e4bd3cdb568b97d3b57ce25c4aa423a776e4c78d33456ef31028dd444224c131"} Jul 31 18:47:11.828027 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:11.827672 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-867bb78f55-dq6bh" event={"ID":"9ebd8658-928b-486a-a597-cb258484c6b1","Type":"ContainerDied","Data":"07d710750f7df10a60ffe4943e868aadf20721d41df9c8e6b26389267f64f5dc"} Jul 31 18:47:11.828027 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:11.827691 2565 scope.go:117] "RemoveContainer" containerID="e4bd3cdb568b97d3b57ce25c4aa423a776e4c78d33456ef31028dd444224c131" Jul 31 18:47:11.838096 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:11.838073 2565 scope.go:117] "RemoveContainer" containerID="e4bd3cdb568b97d3b57ce25c4aa423a776e4c78d33456ef31028dd444224c131" Jul 31 18:47:11.838681 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:47:11.838657 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4bd3cdb568b97d3b57ce25c4aa423a776e4c78d33456ef31028dd444224c131\": container with ID starting with e4bd3cdb568b97d3b57ce25c4aa423a776e4c78d33456ef31028dd444224c131 not found: ID does not exist" containerID="e4bd3cdb568b97d3b57ce25c4aa423a776e4c78d33456ef31028dd444224c131" Jul 31 18:47:11.838783 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:11.838688 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4bd3cdb568b97d3b57ce25c4aa423a776e4c78d33456ef31028dd444224c131"} err="failed to get container status \"e4bd3cdb568b97d3b57ce25c4aa423a776e4c78d33456ef31028dd444224c131\": rpc error: code = NotFound desc = could not find container \"e4bd3cdb568b97d3b57ce25c4aa423a776e4c78d33456ef31028dd444224c131\": container with ID starting with e4bd3cdb568b97d3b57ce25c4aa423a776e4c78d33456ef31028dd444224c131 not found: ID does not exist" Jul 31 18:47:11.860975 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:11.860947 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-867bb78f55-dq6bh"] Jul 31 18:47:11.871984 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:11.871963 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-867bb78f55-dq6bh"] Jul 31 18:47:11.884690 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:11.884665 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ebd8658-928b-486a-a597-cb258484c6b1" path="/var/lib/kubelet/pods/9ebd8658-928b-486a-a597-cb258484c6b1/volumes" Jul 31 18:47:40.733101 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:40.733064 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:47:42.241001 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:42.240954 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:47:51.012119 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:51.012079 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:47:51.111598 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:47:51.111566 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:48:12.667338 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:12.667227 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:48:13.477387 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:13.477350 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:48:14.712017 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:14.711985 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wwq85"] Jul 31 18:48:14.712380 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:14.712252 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ebd8658-928b-486a-a597-cb258484c6b1" containerName="console" Jul 31 18:48:14.712380 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:14.712288 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ebd8658-928b-486a-a597-cb258484c6b1" containerName="console" Jul 31 18:48:14.712380 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:14.712347 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ebd8658-928b-486a-a597-cb258484c6b1" containerName="console" Jul 31 18:48:14.714957 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:14.714921 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wwq85" Jul 31 18:48:14.717534 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:14.717508 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-7spb9\"" Jul 31 18:48:14.718026 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:14.718004 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 31 18:48:14.718809 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:14.718788 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 31 18:48:14.729225 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:14.729203 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wwq85"] Jul 31 18:48:14.817296 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:14.817262 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvz2c\" (UniqueName: \"kubernetes.io/projected/9ff483f0-a408-4cb4-b94e-21a91d42dc4a-kube-api-access-hvz2c\") pod \"cert-manager-operator-controller-manager-55c444bf86-wwq85\" (UID: \"9ff483f0-a408-4cb4-b94e-21a91d42dc4a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wwq85" Jul 31 18:48:14.817467 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:14.817314 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ff483f0-a408-4cb4-b94e-21a91d42dc4a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wwq85\" (UID: \"9ff483f0-a408-4cb4-b94e-21a91d42dc4a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wwq85" Jul 31 18:48:14.918637 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:14.918609 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ff483f0-a408-4cb4-b94e-21a91d42dc4a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wwq85\" (UID: \"9ff483f0-a408-4cb4-b94e-21a91d42dc4a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wwq85" Jul 31 18:48:14.918759 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:14.918668 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvz2c\" (UniqueName: \"kubernetes.io/projected/9ff483f0-a408-4cb4-b94e-21a91d42dc4a-kube-api-access-hvz2c\") pod \"cert-manager-operator-controller-manager-55c444bf86-wwq85\" (UID: \"9ff483f0-a408-4cb4-b94e-21a91d42dc4a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wwq85" Jul 31 18:48:14.918978 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:14.918959 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ff483f0-a408-4cb4-b94e-21a91d42dc4a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wwq85\" (UID: \"9ff483f0-a408-4cb4-b94e-21a91d42dc4a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wwq85" Jul 31 18:48:14.936380 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:14.936354 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvz2c\" (UniqueName: \"kubernetes.io/projected/9ff483f0-a408-4cb4-b94e-21a91d42dc4a-kube-api-access-hvz2c\") pod \"cert-manager-operator-controller-manager-55c444bf86-wwq85\" (UID: \"9ff483f0-a408-4cb4-b94e-21a91d42dc4a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wwq85" Jul 31 18:48:15.024599 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:15.024566 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wwq85" Jul 31 18:48:15.158952 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:15.158918 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wwq85"] Jul 31 18:48:15.161999 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:48:15.161976 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ff483f0_a408_4cb4_b94e_21a91d42dc4a.slice/crio-1246c190a2d0a9588590bdc2a5d95c3fe2f0e41d2bec27fb76004edfdb68060f WatchSource:0}: Error finding container 1246c190a2d0a9588590bdc2a5d95c3fe2f0e41d2bec27fb76004edfdb68060f: Status 404 returned error can't find the container with id 1246c190a2d0a9588590bdc2a5d95c3fe2f0e41d2bec27fb76004edfdb68060f Jul 31 18:48:16.052860 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:16.052820 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wwq85" event={"ID":"9ff483f0-a408-4cb4-b94e-21a91d42dc4a","Type":"ContainerStarted","Data":"1246c190a2d0a9588590bdc2a5d95c3fe2f0e41d2bec27fb76004edfdb68060f"} Jul 31 18:48:16.278311 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:16.278274 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:48:16.378828 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:16.378741 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:48:18.062141 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:18.062108 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wwq85" event={"ID":"9ff483f0-a408-4cb4-b94e-21a91d42dc4a","Type":"ContainerStarted","Data":"e7c2d94bdea97d40e57e817388eb67df5c689a16340d7dd72550db507268c1b9"} Jul 31 18:48:18.085189 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:18.085129 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wwq85" podStartSLOduration=1.58230392 podStartE2EDuration="4.085109912s" totalImagesPullingTime="2.502805992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:48:18.083223091 +0000 UTC m=+274.721212891" watchObservedRunningTime="2026-07-31 18:48:18.085109912 +0000 UTC m=+274.723099713" Jul 31 18:48:18.333401 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:18.333322 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:48:18.465008 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:18.464969 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:48:18.567941 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:18.567908 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:48:20.733591 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:20.733552 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:48:20.950997 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:20.950967 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:48:23.274426 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:23.274391 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:48:23.341446 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:23.341400 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:48:28.553429 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:28.553391 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:48:34.682252 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:34.682213 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:48:34.780775 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:34.780735 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:48:36.706442 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:36.706404 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7kspr"] Jul 31 18:48:36.709345 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:36.709323 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7kspr" Jul 31 18:48:36.711451 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:36.711433 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 31 18:48:36.711559 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:36.711435 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 31 18:48:36.711752 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:36.711738 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-67zx5\"" Jul 31 18:48:36.717980 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:36.717957 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7kspr"] Jul 31 18:48:36.793788 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:36.793762 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x5g7\" (UniqueName: \"kubernetes.io/projected/69612ccc-85d1-4b6e-964e-6aa55d91a327-kube-api-access-7x5g7\") pod \"openshift-lws-operator-fd8ccff4c-7kspr\" (UID: \"69612ccc-85d1-4b6e-964e-6aa55d91a327\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7kspr" Jul 31 18:48:36.793889 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:36.793806 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69612ccc-85d1-4b6e-964e-6aa55d91a327-tmp\") pod \"openshift-lws-operator-fd8ccff4c-7kspr\" (UID: \"69612ccc-85d1-4b6e-964e-6aa55d91a327\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7kspr" Jul 31 18:48:36.894504 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:36.894469 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x5g7\" (UniqueName: \"kubernetes.io/projected/69612ccc-85d1-4b6e-964e-6aa55d91a327-kube-api-access-7x5g7\") pod \"openshift-lws-operator-fd8ccff4c-7kspr\" (UID: \"69612ccc-85d1-4b6e-964e-6aa55d91a327\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7kspr" Jul 31 18:48:36.894655 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:36.894518 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69612ccc-85d1-4b6e-964e-6aa55d91a327-tmp\") pod \"openshift-lws-operator-fd8ccff4c-7kspr\" (UID: \"69612ccc-85d1-4b6e-964e-6aa55d91a327\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7kspr" Jul 31 18:48:36.894880 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:36.894862 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69612ccc-85d1-4b6e-964e-6aa55d91a327-tmp\") pod \"openshift-lws-operator-fd8ccff4c-7kspr\" (UID: \"69612ccc-85d1-4b6e-964e-6aa55d91a327\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7kspr" Jul 31 18:48:36.903686 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:36.903662 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x5g7\" (UniqueName: \"kubernetes.io/projected/69612ccc-85d1-4b6e-964e-6aa55d91a327-kube-api-access-7x5g7\") pod \"openshift-lws-operator-fd8ccff4c-7kspr\" (UID: \"69612ccc-85d1-4b6e-964e-6aa55d91a327\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7kspr" Jul 31 18:48:37.018452 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:37.018417 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7kspr" Jul 31 18:48:37.145233 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:37.145170 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7kspr"] Jul 31 18:48:37.147975 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:48:37.147951 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69612ccc_85d1_4b6e_964e_6aa55d91a327.slice/crio-40e4d47f382d03fbbab7a63b2e17a5741b8255c2807e260acf6810b1857ae30f WatchSource:0}: Error finding container 40e4d47f382d03fbbab7a63b2e17a5741b8255c2807e260acf6810b1857ae30f: Status 404 returned error can't find the container with id 40e4d47f382d03fbbab7a63b2e17a5741b8255c2807e260acf6810b1857ae30f Jul 31 18:48:38.139984 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:38.139945 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7kspr" event={"ID":"69612ccc-85d1-4b6e-964e-6aa55d91a327","Type":"ContainerStarted","Data":"40e4d47f382d03fbbab7a63b2e17a5741b8255c2807e260acf6810b1857ae30f"} Jul 31 18:48:40.149569 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:40.149531 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7kspr" event={"ID":"69612ccc-85d1-4b6e-964e-6aa55d91a327","Type":"ContainerStarted","Data":"f7e27f40911b663649543f6b073c4e52843a3513fbe88ab8be74cb40273f46a2"} Jul 31 18:48:40.170388 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:40.170338 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7kspr" podStartSLOduration=1.7753749349999999 podStartE2EDuration="4.170323925s" totalImagesPullingTime="2.39494899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:48:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:48:40.168437177 +0000 UTC m=+296.806426980" watchObservedRunningTime="2026-07-31 18:48:40.170323925 +0000 UTC m=+296.808313725" Jul 31 18:48:40.876635 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:40.876599 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:48:40.976126 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:40.976079 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:48:42.610973 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:42.610941 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:48:43.556074 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:43.556041 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:48:43.666588 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:43.666559 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:48:43.767378 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:43.767352 2565 kubelet.go:1648] "Image garbage collection succeeded" Jul 31 18:48:55.687205 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:55.687169 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:48:55.799741 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:55.799690 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:48:56.301878 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:56.301841 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:48:56.493527 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:56.493490 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:48:56.598439 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:56.598356 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:48:56.684123 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:56.684088 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-72wp8"] Jul 31 18:48:56.687344 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:56.687325 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-72wp8" Jul 31 18:48:56.690066 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:56.690040 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 31 18:48:56.690172 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:56.690105 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 31 18:48:56.690172 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:56.690144 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 31 18:48:56.691884 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:56.691867 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-x6fdg\"" Jul 31 18:48:56.692311 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:56.692297 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 31 18:48:56.704665 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:56.704641 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-72wp8"] Jul 31 18:48:56.757166 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:56.757141 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9eca874d-935c-4e8f-a053-dd892772587f-tmp\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-72wp8\" (UID: \"9eca874d-935c-4e8f-a053-dd892772587f\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-72wp8" Jul 31 18:48:56.757341 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:56.757179 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9eca874d-935c-4e8f-a053-dd892772587f-webhook-cert\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-72wp8\" (UID: \"9eca874d-935c-4e8f-a053-dd892772587f\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-72wp8" Jul 31 18:48:56.757341 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:56.757204 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgwzr\" (UniqueName: \"kubernetes.io/projected/9eca874d-935c-4e8f-a053-dd892772587f-kube-api-access-vgwzr\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-72wp8\" (UID: \"9eca874d-935c-4e8f-a053-dd892772587f\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-72wp8" Jul 31 18:48:56.757341 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:56.757280 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9eca874d-935c-4e8f-a053-dd892772587f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-72wp8\" (UID: \"9eca874d-935c-4e8f-a053-dd892772587f\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-72wp8" Jul 31 18:48:56.757341 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:56.757304 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9eca874d-935c-4e8f-a053-dd892772587f-manifests\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-72wp8\" (UID: \"9eca874d-935c-4e8f-a053-dd892772587f\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-72wp8" Jul 31 18:48:56.847674 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:56.847638 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:48:56.857682 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:56.857612 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9eca874d-935c-4e8f-a053-dd892772587f-tmp\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-72wp8\" (UID: \"9eca874d-935c-4e8f-a053-dd892772587f\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-72wp8" Jul 31 18:48:56.857682 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:56.857655 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9eca874d-935c-4e8f-a053-dd892772587f-webhook-cert\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-72wp8\" (UID: \"9eca874d-935c-4e8f-a053-dd892772587f\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-72wp8" Jul 31 18:48:56.857682 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:56.857675 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgwzr\" (UniqueName: \"kubernetes.io/projected/9eca874d-935c-4e8f-a053-dd892772587f-kube-api-access-vgwzr\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-72wp8\" (UID: \"9eca874d-935c-4e8f-a053-dd892772587f\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-72wp8" Jul 31 18:48:56.857854 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:56.857707 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9eca874d-935c-4e8f-a053-dd892772587f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-72wp8\" (UID: \"9eca874d-935c-4e8f-a053-dd892772587f\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-72wp8" Jul 31 18:48:56.857854 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:56.857727 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9eca874d-935c-4e8f-a053-dd892772587f-manifests\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-72wp8\" (UID: \"9eca874d-935c-4e8f-a053-dd892772587f\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-72wp8" Jul 31 18:48:56.858135 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:56.858106 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9eca874d-935c-4e8f-a053-dd892772587f-manifests\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-72wp8\" (UID: \"9eca874d-935c-4e8f-a053-dd892772587f\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-72wp8" Jul 31 18:48:56.859989 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:56.859960 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9eca874d-935c-4e8f-a053-dd892772587f-tmp\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-72wp8\" (UID: \"9eca874d-935c-4e8f-a053-dd892772587f\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-72wp8" Jul 31 18:48:56.860151 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:56.860132 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9eca874d-935c-4e8f-a053-dd892772587f-webhook-cert\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-72wp8\" (UID: \"9eca874d-935c-4e8f-a053-dd892772587f\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-72wp8" Jul 31 18:48:56.860197 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:56.860170 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9eca874d-935c-4e8f-a053-dd892772587f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-72wp8\" (UID: \"9eca874d-935c-4e8f-a053-dd892772587f\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-72wp8" Jul 31 18:48:56.873925 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:56.873901 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgwzr\" (UniqueName: \"kubernetes.io/projected/9eca874d-935c-4e8f-a053-dd892772587f-kube-api-access-vgwzr\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-72wp8\" (UID: \"9eca874d-935c-4e8f-a053-dd892772587f\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-72wp8" Jul 31 18:48:56.956335 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:56.956291 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:48:56.997553 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:56.997527 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-72wp8" Jul 31 18:48:57.061415 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:57.061364 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:48:57.154187 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:57.154109 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-72wp8"] Jul 31 18:48:57.157192 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:48:57.157166 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eca874d_935c_4e8f_a053_dd892772587f.slice/crio-4ed920e1999f4932e4254d868304fedf1acdce07b14103ed9e0d1347e350a525 WatchSource:0}: Error finding container 4ed920e1999f4932e4254d868304fedf1acdce07b14103ed9e0d1347e350a525: Status 404 returned error can't find the container with id 4ed920e1999f4932e4254d868304fedf1acdce07b14103ed9e0d1347e350a525 Jul 31 18:48:57.159033 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:57.159017 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 18:48:57.216853 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:57.216819 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-72wp8" event={"ID":"9eca874d-935c-4e8f-a053-dd892772587f","Type":"ContainerStarted","Data":"4ed920e1999f4932e4254d868304fedf1acdce07b14103ed9e0d1347e350a525"} Jul 31 18:48:57.257522 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:57.257485 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:48:57.366904 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:57.366868 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:48:57.472089 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:57.472001 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:48:58.866663 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:58.866622 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:48:58.973700 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:48:58.973663 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:49:01.236526 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:01.236484 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-72wp8" event={"ID":"9eca874d-935c-4e8f-a053-dd892772587f","Type":"ContainerStarted","Data":"0fc57990f5369d5b62076ce15791474bc0f401d0b49dcd5025e5a43e70e46549"} Jul 31 18:49:02.241700 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:02.241664 2565 generic.go:356] "Generic (PLEG): container finished" podID="9eca874d-935c-4e8f-a053-dd892772587f" containerID="0fc57990f5369d5b62076ce15791474bc0f401d0b49dcd5025e5a43e70e46549" exitCode=0 Jul 31 18:49:02.242075 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:02.241712 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-72wp8" event={"ID":"9eca874d-935c-4e8f-a053-dd892772587f","Type":"ContainerDied","Data":"0fc57990f5369d5b62076ce15791474bc0f401d0b49dcd5025e5a43e70e46549"} Jul 31 18:49:03.247384 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:03.247344 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-72wp8" event={"ID":"9eca874d-935c-4e8f-a053-dd892772587f","Type":"ContainerStarted","Data":"d08772002b6194b3bd6523d5290db762c9a2b14803c9156f16cd5616d7652a33"} Jul 31 18:49:03.247855 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:03.247481 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-72wp8" Jul 31 18:49:03.270203 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:03.270151 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-72wp8" podStartSLOduration=2.79130764 podStartE2EDuration="7.270137223s" totalImagesPullingTime="3.393255242s" totalInitContainerRuntime="1.085574341s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:48:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:49:03.268292821 +0000 UTC m=+319.906282621" watchObservedRunningTime="2026-07-31 18:49:03.270137223 +0000 UTC m=+319.908127023" Jul 31 18:49:08.128110 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:08.128064 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:49:09.716879 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:09.716839 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:49:14.254094 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:14.254062 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-72wp8" Jul 31 18:49:15.956735 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:15.956695 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:49:16.094795 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:16.094761 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:49:16.619043 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:16.619012 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:49:16.691611 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:16.691578 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-86dbc777-nbl8z"] Jul 31 18:49:16.699348 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:16.699329 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86dbc777-nbl8z" Jul 31 18:49:16.704709 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:16.704688 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 31 18:49:16.704796 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:16.704720 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 31 18:49:16.709369 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:16.709352 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-gqjql\"" Jul 31 18:49:16.723266 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:16.723231 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86dbc777-nbl8z"] Jul 31 18:49:16.822443 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:16.822412 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b56a1d6-fe42-4319-8e31-90bc4def9285-tmp\") pod \"kube-auth-proxy-86dbc777-nbl8z\" (UID: \"1b56a1d6-fe42-4319-8e31-90bc4def9285\") " pod="openshift-ingress/kube-auth-proxy-86dbc777-nbl8z" Jul 31 18:49:16.822588 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:16.822446 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7629d\" (UniqueName: \"kubernetes.io/projected/1b56a1d6-fe42-4319-8e31-90bc4def9285-kube-api-access-7629d\") pod \"kube-auth-proxy-86dbc777-nbl8z\" (UID: \"1b56a1d6-fe42-4319-8e31-90bc4def9285\") " pod="openshift-ingress/kube-auth-proxy-86dbc777-nbl8z" Jul 31 18:49:16.822588 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:16.822507 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b56a1d6-fe42-4319-8e31-90bc4def9285-tls-certs\") pod \"kube-auth-proxy-86dbc777-nbl8z\" (UID: \"1b56a1d6-fe42-4319-8e31-90bc4def9285\") " pod="openshift-ingress/kube-auth-proxy-86dbc777-nbl8z" Jul 31 18:49:16.866926 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:16.866900 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-ktlkc"] Jul 31 18:49:16.869072 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:16.869034 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-ktlkc" Jul 31 18:49:16.871228 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:16.871203 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-rss76\"" Jul 31 18:49:16.872027 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:16.872010 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 31 18:49:16.885693 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:16.885657 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-ktlkc"] Jul 31 18:49:16.923349 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:16.923318 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b56a1d6-fe42-4319-8e31-90bc4def9285-tmp\") pod \"kube-auth-proxy-86dbc777-nbl8z\" (UID: \"1b56a1d6-fe42-4319-8e31-90bc4def9285\") " pod="openshift-ingress/kube-auth-proxy-86dbc777-nbl8z" Jul 31 18:49:16.923349 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:16.923352 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7629d\" (UniqueName: \"kubernetes.io/projected/1b56a1d6-fe42-4319-8e31-90bc4def9285-kube-api-access-7629d\") pod \"kube-auth-proxy-86dbc777-nbl8z\" (UID: \"1b56a1d6-fe42-4319-8e31-90bc4def9285\") " pod="openshift-ingress/kube-auth-proxy-86dbc777-nbl8z" Jul 31 18:49:16.923562 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:16.923398 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b56a1d6-fe42-4319-8e31-90bc4def9285-tls-certs\") pod \"kube-auth-proxy-86dbc777-nbl8z\" (UID: \"1b56a1d6-fe42-4319-8e31-90bc4def9285\") " pod="openshift-ingress/kube-auth-proxy-86dbc777-nbl8z" Jul 31 18:49:16.923562 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:49:16.923500 2565 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 31 18:49:16.923562 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:49:16.923561 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b56a1d6-fe42-4319-8e31-90bc4def9285-tls-certs podName:1b56a1d6-fe42-4319-8e31-90bc4def9285 nodeName:}" failed. No retries permitted until 2026-07-31 18:49:17.423545976 +0000 UTC m=+334.061535754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/1b56a1d6-fe42-4319-8e31-90bc4def9285-tls-certs") pod "kube-auth-proxy-86dbc777-nbl8z" (UID: "1b56a1d6-fe42-4319-8e31-90bc4def9285") : secret "kube-auth-proxy-tls" not found Jul 31 18:49:16.925611 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:16.925584 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b56a1d6-fe42-4319-8e31-90bc4def9285-tmp\") pod \"kube-auth-proxy-86dbc777-nbl8z\" (UID: \"1b56a1d6-fe42-4319-8e31-90bc4def9285\") " pod="openshift-ingress/kube-auth-proxy-86dbc777-nbl8z" Jul 31 18:49:16.933230 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:16.933192 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:49:16.937911 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:16.937885 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7629d\" (UniqueName: \"kubernetes.io/projected/1b56a1d6-fe42-4319-8e31-90bc4def9285-kube-api-access-7629d\") pod \"kube-auth-proxy-86dbc777-nbl8z\" (UID: \"1b56a1d6-fe42-4319-8e31-90bc4def9285\") " pod="openshift-ingress/kube-auth-proxy-86dbc777-nbl8z" Jul 31 18:49:17.024570 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:17.024531 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tgg7\" (UniqueName: \"kubernetes.io/projected/2b6c0760-f3ae-46b2-911a-c1e7e2663485-kube-api-access-2tgg7\") pod \"odh-model-controller-7998c67dc4-ktlkc\" (UID: \"2b6c0760-f3ae-46b2-911a-c1e7e2663485\") " pod="opendatahub/odh-model-controller-7998c67dc4-ktlkc" Jul 31 18:49:17.024914 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:17.024682 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b6c0760-f3ae-46b2-911a-c1e7e2663485-cert\") pod \"odh-model-controller-7998c67dc4-ktlkc\" (UID: \"2b6c0760-f3ae-46b2-911a-c1e7e2663485\") " pod="opendatahub/odh-model-controller-7998c67dc4-ktlkc" Jul 31 18:49:17.106675 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:17.106618 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:49:17.125313 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:17.125228 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b6c0760-f3ae-46b2-911a-c1e7e2663485-cert\") pod \"odh-model-controller-7998c67dc4-ktlkc\" (UID: \"2b6c0760-f3ae-46b2-911a-c1e7e2663485\") " pod="opendatahub/odh-model-controller-7998c67dc4-ktlkc" Jul 31 18:49:17.125420 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:17.125310 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tgg7\" (UniqueName: \"kubernetes.io/projected/2b6c0760-f3ae-46b2-911a-c1e7e2663485-kube-api-access-2tgg7\") pod \"odh-model-controller-7998c67dc4-ktlkc\" (UID: \"2b6c0760-f3ae-46b2-911a-c1e7e2663485\") " pod="opendatahub/odh-model-controller-7998c67dc4-ktlkc" Jul 31 18:49:17.125420 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:49:17.125389 2565 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 31 18:49:17.125491 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:49:17.125446 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b6c0760-f3ae-46b2-911a-c1e7e2663485-cert podName:2b6c0760-f3ae-46b2-911a-c1e7e2663485 nodeName:}" failed. No retries permitted until 2026-07-31 18:49:17.625430485 +0000 UTC m=+334.263420277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2b6c0760-f3ae-46b2-911a-c1e7e2663485-cert") pod "odh-model-controller-7998c67dc4-ktlkc" (UID: "2b6c0760-f3ae-46b2-911a-c1e7e2663485") : secret "odh-model-controller-webhook-cert" not found Jul 31 18:49:17.136185 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:17.136159 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tgg7\" (UniqueName: \"kubernetes.io/projected/2b6c0760-f3ae-46b2-911a-c1e7e2663485-kube-api-access-2tgg7\") pod \"odh-model-controller-7998c67dc4-ktlkc\" (UID: \"2b6c0760-f3ae-46b2-911a-c1e7e2663485\") " pod="opendatahub/odh-model-controller-7998c67dc4-ktlkc" Jul 31 18:49:17.295887 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:17.295857 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:49:17.414307 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:17.414201 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:49:17.428331 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:17.428306 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b56a1d6-fe42-4319-8e31-90bc4def9285-tls-certs\") pod \"kube-auth-proxy-86dbc777-nbl8z\" (UID: \"1b56a1d6-fe42-4319-8e31-90bc4def9285\") " pod="openshift-ingress/kube-auth-proxy-86dbc777-nbl8z" Jul 31 18:49:17.430642 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:17.430622 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b56a1d6-fe42-4319-8e31-90bc4def9285-tls-certs\") pod \"kube-auth-proxy-86dbc777-nbl8z\" (UID: \"1b56a1d6-fe42-4319-8e31-90bc4def9285\") " pod="openshift-ingress/kube-auth-proxy-86dbc777-nbl8z" Jul 31 18:49:17.516163 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:17.516130 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:49:17.608305 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:17.608274 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86dbc777-nbl8z" Jul 31 18:49:17.630439 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:17.630412 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b6c0760-f3ae-46b2-911a-c1e7e2663485-cert\") pod \"odh-model-controller-7998c67dc4-ktlkc\" (UID: \"2b6c0760-f3ae-46b2-911a-c1e7e2663485\") " pod="opendatahub/odh-model-controller-7998c67dc4-ktlkc" Jul 31 18:49:17.632727 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:17.632703 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b6c0760-f3ae-46b2-911a-c1e7e2663485-cert\") pod \"odh-model-controller-7998c67dc4-ktlkc\" (UID: \"2b6c0760-f3ae-46b2-911a-c1e7e2663485\") " pod="opendatahub/odh-model-controller-7998c67dc4-ktlkc" Jul 31 18:49:17.745395 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:17.745367 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86dbc777-nbl8z"] Jul 31 18:49:17.748542 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:49:17.748518 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b56a1d6_fe42_4319_8e31_90bc4def9285.slice/crio-58ce7ca529cc142abb3eeef93873feff25b2b6cda26a57494b39a276f50720db WatchSource:0}: Error finding container 58ce7ca529cc142abb3eeef93873feff25b2b6cda26a57494b39a276f50720db: Status 404 returned error can't find the container with id 58ce7ca529cc142abb3eeef93873feff25b2b6cda26a57494b39a276f50720db Jul 31 18:49:17.779069 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:17.779049 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-ktlkc" Jul 31 18:49:17.915583 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:17.915418 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-ktlkc"] Jul 31 18:49:17.918053 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:49:17.918027 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6c0760_f3ae_46b2_911a_c1e7e2663485.slice/crio-5e70a52858de11a289a44e683cc90b31731fd2fd5a7afa49133c13c606d4a7c3 WatchSource:0}: Error finding container 5e70a52858de11a289a44e683cc90b31731fd2fd5a7afa49133c13c606d4a7c3: Status 404 returned error can't find the container with id 5e70a52858de11a289a44e683cc90b31731fd2fd5a7afa49133c13c606d4a7c3 Jul 31 18:49:18.073133 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.073104 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:49:18.157414 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.157379 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-wnvkm"] Jul 31 18:49:18.160128 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.160113 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-wnvkm" Jul 31 18:49:18.162175 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.162152 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-m55g4\"" Jul 31 18:49:18.162320 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.162222 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 31 18:49:18.172110 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.172085 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-wnvkm"] Jul 31 18:49:18.173888 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.173861 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:49:18.237820 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.237792 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h"] Jul 31 18:49:18.239976 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.239958 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" Jul 31 18:49:18.241766 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.241748 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 31 18:49:18.241873 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.241780 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 31 18:49:18.241924 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.241905 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 31 18:49:18.242024 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.242011 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 31 18:49:18.242168 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.242149 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qtxwn\"" Jul 31 18:49:18.253754 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.253735 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h"] Jul 31 18:49:18.281550 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.281525 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:49:18.306964 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.306929 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-ktlkc" event={"ID":"2b6c0760-f3ae-46b2-911a-c1e7e2663485","Type":"ContainerStarted","Data":"5e70a52858de11a289a44e683cc90b31731fd2fd5a7afa49133c13c606d4a7c3"} Jul 31 18:49:18.308195 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.308170 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86dbc777-nbl8z" event={"ID":"1b56a1d6-fe42-4319-8e31-90bc4def9285","Type":"ContainerStarted","Data":"58ce7ca529cc142abb3eeef93873feff25b2b6cda26a57494b39a276f50720db"} Jul 31 18:49:18.337608 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.337546 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e034bfa6-7b57-4146-9682-508d276a6eb2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-vgs4h\" (UID: \"e034bfa6-7b57-4146-9682-508d276a6eb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" Jul 31 18:49:18.337608 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.337590 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e034bfa6-7b57-4146-9682-508d276a6eb2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-vgs4h\" (UID: \"e034bfa6-7b57-4146-9682-508d276a6eb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" Jul 31 18:49:18.337608 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.337606 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg6dw\" (UniqueName: \"kubernetes.io/projected/e034bfa6-7b57-4146-9682-508d276a6eb2-kube-api-access-qg6dw\") pod \"istiod-openshift-gateway-7b7995d5cd-vgs4h\" (UID: \"e034bfa6-7b57-4146-9682-508d276a6eb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" Jul 31 18:49:18.337786 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.337660 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e034bfa6-7b57-4146-9682-508d276a6eb2-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-vgs4h\" (UID: \"e034bfa6-7b57-4146-9682-508d276a6eb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" Jul 31 18:49:18.337786 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.337680 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e034bfa6-7b57-4146-9682-508d276a6eb2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-vgs4h\" (UID: \"e034bfa6-7b57-4146-9682-508d276a6eb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" Jul 31 18:49:18.337786 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.337700 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bec16275-6e1a-4d0e-84a8-b648f317611a-tls-certs\") pod \"model-serving-api-54d89bd5fc-wnvkm\" (UID: \"bec16275-6e1a-4d0e-84a8-b648f317611a\") " pod="opendatahub/model-serving-api-54d89bd5fc-wnvkm" Jul 31 18:49:18.337786 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.337716 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzdxc\" (UniqueName: \"kubernetes.io/projected/bec16275-6e1a-4d0e-84a8-b648f317611a-kube-api-access-hzdxc\") pod \"model-serving-api-54d89bd5fc-wnvkm\" (UID: \"bec16275-6e1a-4d0e-84a8-b648f317611a\") " pod="opendatahub/model-serving-api-54d89bd5fc-wnvkm" Jul 31 18:49:18.337786 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.337738 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e034bfa6-7b57-4146-9682-508d276a6eb2-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-vgs4h\" (UID: \"e034bfa6-7b57-4146-9682-508d276a6eb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" Jul 31 18:49:18.337786 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.337765 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e034bfa6-7b57-4146-9682-508d276a6eb2-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-vgs4h\" (UID: \"e034bfa6-7b57-4146-9682-508d276a6eb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" Jul 31 18:49:18.379511 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.379472 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:49:18.438619 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.438589 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e034bfa6-7b57-4146-9682-508d276a6eb2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-vgs4h\" (UID: \"e034bfa6-7b57-4146-9682-508d276a6eb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" Jul 31 18:49:18.438769 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.438637 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e034bfa6-7b57-4146-9682-508d276a6eb2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-vgs4h\" (UID: \"e034bfa6-7b57-4146-9682-508d276a6eb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" Jul 31 18:49:18.438769 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.438655 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg6dw\" (UniqueName: \"kubernetes.io/projected/e034bfa6-7b57-4146-9682-508d276a6eb2-kube-api-access-qg6dw\") pod \"istiod-openshift-gateway-7b7995d5cd-vgs4h\" (UID: \"e034bfa6-7b57-4146-9682-508d276a6eb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" Jul 31 18:49:18.438769 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.438680 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e034bfa6-7b57-4146-9682-508d276a6eb2-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-vgs4h\" (UID: \"e034bfa6-7b57-4146-9682-508d276a6eb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" Jul 31 18:49:18.438769 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.438707 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e034bfa6-7b57-4146-9682-508d276a6eb2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-vgs4h\" (UID: \"e034bfa6-7b57-4146-9682-508d276a6eb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" Jul 31 18:49:18.438769 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.438725 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bec16275-6e1a-4d0e-84a8-b648f317611a-tls-certs\") pod \"model-serving-api-54d89bd5fc-wnvkm\" (UID: \"bec16275-6e1a-4d0e-84a8-b648f317611a\") " pod="opendatahub/model-serving-api-54d89bd5fc-wnvkm" Jul 31 18:49:18.438769 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.438744 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzdxc\" (UniqueName: \"kubernetes.io/projected/bec16275-6e1a-4d0e-84a8-b648f317611a-kube-api-access-hzdxc\") pod \"model-serving-api-54d89bd5fc-wnvkm\" (UID: \"bec16275-6e1a-4d0e-84a8-b648f317611a\") " pod="opendatahub/model-serving-api-54d89bd5fc-wnvkm" Jul 31 18:49:18.439083 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.438776 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e034bfa6-7b57-4146-9682-508d276a6eb2-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-vgs4h\" (UID: \"e034bfa6-7b57-4146-9682-508d276a6eb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" Jul 31 18:49:18.439083 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.438806 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e034bfa6-7b57-4146-9682-508d276a6eb2-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-vgs4h\" (UID: \"e034bfa6-7b57-4146-9682-508d276a6eb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" Jul 31 18:49:18.439501 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.439471 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e034bfa6-7b57-4146-9682-508d276a6eb2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-vgs4h\" (UID: \"e034bfa6-7b57-4146-9682-508d276a6eb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" Jul 31 18:49:18.441376 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.441355 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e034bfa6-7b57-4146-9682-508d276a6eb2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-vgs4h\" (UID: \"e034bfa6-7b57-4146-9682-508d276a6eb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" Jul 31 18:49:18.441478 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.441423 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e034bfa6-7b57-4146-9682-508d276a6eb2-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-vgs4h\" (UID: \"e034bfa6-7b57-4146-9682-508d276a6eb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" Jul 31 18:49:18.441552 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.441536 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e034bfa6-7b57-4146-9682-508d276a6eb2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-vgs4h\" (UID: \"e034bfa6-7b57-4146-9682-508d276a6eb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" Jul 31 18:49:18.441621 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.441602 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bec16275-6e1a-4d0e-84a8-b648f317611a-tls-certs\") pod \"model-serving-api-54d89bd5fc-wnvkm\" (UID: \"bec16275-6e1a-4d0e-84a8-b648f317611a\") " pod="opendatahub/model-serving-api-54d89bd5fc-wnvkm" Jul 31 18:49:18.441770 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.441754 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e034bfa6-7b57-4146-9682-508d276a6eb2-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-vgs4h\" (UID: \"e034bfa6-7b57-4146-9682-508d276a6eb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" Jul 31 18:49:18.451289 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.451267 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzdxc\" (UniqueName: \"kubernetes.io/projected/bec16275-6e1a-4d0e-84a8-b648f317611a-kube-api-access-hzdxc\") pod \"model-serving-api-54d89bd5fc-wnvkm\" (UID: \"bec16275-6e1a-4d0e-84a8-b648f317611a\") " pod="opendatahub/model-serving-api-54d89bd5fc-wnvkm" Jul 31 18:49:18.452344 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.452324 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e034bfa6-7b57-4146-9682-508d276a6eb2-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-vgs4h\" (UID: \"e034bfa6-7b57-4146-9682-508d276a6eb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" Jul 31 18:49:18.452527 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.452509 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg6dw\" (UniqueName: \"kubernetes.io/projected/e034bfa6-7b57-4146-9682-508d276a6eb2-kube-api-access-qg6dw\") pod \"istiod-openshift-gateway-7b7995d5cd-vgs4h\" (UID: \"e034bfa6-7b57-4146-9682-508d276a6eb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" Jul 31 18:49:18.470409 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.470390 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-wnvkm" Jul 31 18:49:18.548970 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.548940 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" Jul 31 18:49:18.600401 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.600373 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-wnvkm"] Jul 31 18:49:18.604115 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:49:18.604088 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbec16275_6e1a_4d0e_84a8_b648f317611a.slice/crio-6a960b30495ef35a8e1582aa10925e50eaed7f7b588453d05845cf992b9f7fe7 WatchSource:0}: Error finding container 6a960b30495ef35a8e1582aa10925e50eaed7f7b588453d05845cf992b9f7fe7: Status 404 returned error can't find the container with id 6a960b30495ef35a8e1582aa10925e50eaed7f7b588453d05845cf992b9f7fe7 Jul 31 18:49:18.691884 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:18.691708 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h"] Jul 31 18:49:18.694407 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:49:18.694362 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode034bfa6_7b57_4146_9682_508d276a6eb2.slice/crio-baa1e2ce9ce536bb12a5e512e505817fcc33bac952bafba3b8801066cabda752 WatchSource:0}: Error finding container baa1e2ce9ce536bb12a5e512e505817fcc33bac952bafba3b8801066cabda752: Status 404 returned error can't find the container with id baa1e2ce9ce536bb12a5e512e505817fcc33bac952bafba3b8801066cabda752 Jul 31 18:49:19.318789 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:19.318749 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-wnvkm" event={"ID":"bec16275-6e1a-4d0e-84a8-b648f317611a","Type":"ContainerStarted","Data":"6a960b30495ef35a8e1582aa10925e50eaed7f7b588453d05845cf992b9f7fe7"} Jul 31 18:49:19.320930 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:19.320897 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" event={"ID":"e034bfa6-7b57-4146-9682-508d276a6eb2","Type":"ContainerStarted","Data":"baa1e2ce9ce536bb12a5e512e505817fcc33bac952bafba3b8801066cabda752"} Jul 31 18:49:19.381422 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:19.381385 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:49:23.219298 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:23.219219 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 18:49:23.219634 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:23.219307 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 18:49:23.343996 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:23.343952 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-ktlkc" event={"ID":"2b6c0760-f3ae-46b2-911a-c1e7e2663485","Type":"ContainerStarted","Data":"dae31060ba923105deeafb0b57ee5a85ec827421fc974e79a3312927b36cc440"} Jul 31 18:49:23.344562 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:23.344498 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-ktlkc" Jul 31 18:49:23.346371 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:23.345977 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86dbc777-nbl8z" event={"ID":"1b56a1d6-fe42-4319-8e31-90bc4def9285","Type":"ContainerStarted","Data":"931a7376ed923c6f5a16ac4d5dbc4aa2bef703c6a528e42eca922b4fe0db4ffa"} Jul 31 18:49:23.348137 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:23.348109 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-wnvkm" event={"ID":"bec16275-6e1a-4d0e-84a8-b648f317611a","Type":"ContainerStarted","Data":"e6021b991fb9448091fcb1f7c86bbe17e1fef68e1265f08e22f0d26bed342bd2"} Jul 31 18:49:23.348492 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:23.348448 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-wnvkm" Jul 31 18:49:23.350033 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:23.350008 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" event={"ID":"e034bfa6-7b57-4146-9682-508d276a6eb2","Type":"ContainerStarted","Data":"1953c4cd9d0b5be3b7e1e90ccd6dd9da68c526d915c8c309528204d1ae3de81b"} Jul 31 18:49:23.350335 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:23.350282 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" Jul 31 18:49:23.367103 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:23.367060 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-ktlkc" podStartSLOduration=2.067616937 podStartE2EDuration="7.367046904s" totalImagesPullingTime="5.299429967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:49:23.365615993 +0000 UTC m=+340.003605794" watchObservedRunningTime="2026-07-31 18:49:23.367046904 +0000 UTC m=+340.005036703" Jul 31 18:49:23.385939 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:23.385898 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-wnvkm" podStartSLOduration=0.779700822 podStartE2EDuration="5.385885775s" totalImagesPullingTime="4.606184953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:49:23.385397274 +0000 UTC m=+340.023387096" watchObservedRunningTime="2026-07-31 18:49:23.385885775 +0000 UTC m=+340.023875575" Jul 31 18:49:23.409768 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:23.409721 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" podStartSLOduration=0.887779075 podStartE2EDuration="5.409707493s" totalImagesPullingTime="4.521928418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:49:23.407547129 +0000 UTC m=+340.045536929" watchObservedRunningTime="2026-07-31 18:49:23.409707493 +0000 UTC m=+340.047697293" Jul 31 18:49:23.428577 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:23.428484 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-86dbc777-nbl8z" podStartSLOduration=1.965704822 podStartE2EDuration="7.428466763s" totalImagesPullingTime="5.462761941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:49:23.428331697 +0000 UTC m=+340.066321503" watchObservedRunningTime="2026-07-31 18:49:23.428466763 +0000 UTC m=+340.066456565" Jul 31 18:49:24.321529 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:24.321486 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:49:24.356824 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:24.356786 2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-vgs4h container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 31 18:49:24.356983 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:24.356849 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" podUID="e034bfa6-7b57-4146-9682-508d276a6eb2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 31 18:49:24.437289 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:24.437231 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:49:25.063394 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.063364 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d"] Jul 31 18:49:25.066728 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.066709 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:25.069440 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.069417 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-5f4p7\"" Jul 31 18:49:25.081855 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.081833 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d"] Jul 31 18:49:25.105349 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.105319 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:49:25.201288 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.201233 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb55cf00-4290-49f4-8367-1d2d69a90016-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d\" (UID: \"eb55cf00-4290-49f4-8367-1d2d69a90016\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:25.201288 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.201289 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wlx9\" (UniqueName: \"kubernetes.io/projected/eb55cf00-4290-49f4-8367-1d2d69a90016-kube-api-access-5wlx9\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d\" (UID: \"eb55cf00-4290-49f4-8367-1d2d69a90016\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:25.201456 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.201363 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb55cf00-4290-49f4-8367-1d2d69a90016-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d\" (UID: \"eb55cf00-4290-49f4-8367-1d2d69a90016\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:25.201456 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.201392 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb55cf00-4290-49f4-8367-1d2d69a90016-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d\" (UID: \"eb55cf00-4290-49f4-8367-1d2d69a90016\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:25.201456 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.201413 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb55cf00-4290-49f4-8367-1d2d69a90016-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d\" (UID: \"eb55cf00-4290-49f4-8367-1d2d69a90016\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:25.201456 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.201439 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb55cf00-4290-49f4-8367-1d2d69a90016-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d\" (UID: \"eb55cf00-4290-49f4-8367-1d2d69a90016\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:25.201627 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.201489 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb55cf00-4290-49f4-8367-1d2d69a90016-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d\" (UID: \"eb55cf00-4290-49f4-8367-1d2d69a90016\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:25.201627 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.201518 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb55cf00-4290-49f4-8367-1d2d69a90016-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d\" (UID: \"eb55cf00-4290-49f4-8367-1d2d69a90016\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:25.201720 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.201620 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb55cf00-4290-49f4-8367-1d2d69a90016-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d\" (UID: \"eb55cf00-4290-49f4-8367-1d2d69a90016\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:25.294233 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.294201 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:49:25.302307 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.302277 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb55cf00-4290-49f4-8367-1d2d69a90016-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d\" (UID: \"eb55cf00-4290-49f4-8367-1d2d69a90016\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:25.302428 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.302314 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb55cf00-4290-49f4-8367-1d2d69a90016-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d\" (UID: \"eb55cf00-4290-49f4-8367-1d2d69a90016\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:25.302428 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.302336 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb55cf00-4290-49f4-8367-1d2d69a90016-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d\" (UID: \"eb55cf00-4290-49f4-8367-1d2d69a90016\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:25.302428 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.302402 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb55cf00-4290-49f4-8367-1d2d69a90016-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d\" (UID: \"eb55cf00-4290-49f4-8367-1d2d69a90016\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:25.302540 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.302436 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb55cf00-4290-49f4-8367-1d2d69a90016-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d\" (UID: \"eb55cf00-4290-49f4-8367-1d2d69a90016\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:25.302540 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.302461 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb55cf00-4290-49f4-8367-1d2d69a90016-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d\" (UID: \"eb55cf00-4290-49f4-8367-1d2d69a90016\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:25.302626 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.302573 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb55cf00-4290-49f4-8367-1d2d69a90016-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d\" (UID: \"eb55cf00-4290-49f4-8367-1d2d69a90016\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:25.302669 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.302649 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb55cf00-4290-49f4-8367-1d2d69a90016-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d\" (UID: \"eb55cf00-4290-49f4-8367-1d2d69a90016\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:25.302719 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.302675 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wlx9\" (UniqueName: \"kubernetes.io/projected/eb55cf00-4290-49f4-8367-1d2d69a90016-kube-api-access-5wlx9\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d\" (UID: \"eb55cf00-4290-49f4-8367-1d2d69a90016\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:25.302765 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.302712 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb55cf00-4290-49f4-8367-1d2d69a90016-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d\" (UID: \"eb55cf00-4290-49f4-8367-1d2d69a90016\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:25.302864 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.302839 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb55cf00-4290-49f4-8367-1d2d69a90016-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d\" (UID: \"eb55cf00-4290-49f4-8367-1d2d69a90016\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:25.302952 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.302933 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb55cf00-4290-49f4-8367-1d2d69a90016-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d\" (UID: \"eb55cf00-4290-49f4-8367-1d2d69a90016\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:25.303004 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.302960 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb55cf00-4290-49f4-8367-1d2d69a90016-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d\" (UID: \"eb55cf00-4290-49f4-8367-1d2d69a90016\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:25.303343 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.303320 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb55cf00-4290-49f4-8367-1d2d69a90016-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d\" (UID: \"eb55cf00-4290-49f4-8367-1d2d69a90016\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:25.304701 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.304680 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb55cf00-4290-49f4-8367-1d2d69a90016-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d\" (UID: \"eb55cf00-4290-49f4-8367-1d2d69a90016\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:25.305019 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.305004 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb55cf00-4290-49f4-8367-1d2d69a90016-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d\" (UID: \"eb55cf00-4290-49f4-8367-1d2d69a90016\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:25.315553 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.315497 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb55cf00-4290-49f4-8367-1d2d69a90016-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d\" (UID: \"eb55cf00-4290-49f4-8367-1d2d69a90016\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:25.315904 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.315881 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wlx9\" (UniqueName: \"kubernetes.io/projected/eb55cf00-4290-49f4-8367-1d2d69a90016-kube-api-access-5wlx9\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d\" (UID: \"eb55cf00-4290-49f4-8367-1d2d69a90016\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:25.377297 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.377274 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:25.385273 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.385225 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:49:25.515348 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:25.515321 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d"] Jul 31 18:49:25.517026 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:49:25.516998 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb55cf00_4290_49f4_8367_1d2d69a90016.slice/crio-99a3dc60ae42b989eda0fcad07f597f448315850edbf67f5d78eba47f4769901 WatchSource:0}: Error finding container 99a3dc60ae42b989eda0fcad07f597f448315850edbf67f5d78eba47f4769901: Status 404 returned error can't find the container with id 99a3dc60ae42b989eda0fcad07f597f448315850edbf67f5d78eba47f4769901 Jul 31 18:49:26.365271 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:26.365218 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" event={"ID":"eb55cf00-4290-49f4-8367-1d2d69a90016","Type":"ContainerStarted","Data":"99a3dc60ae42b989eda0fcad07f597f448315850edbf67f5d78eba47f4769901"} Jul 31 18:49:26.918471 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:26.918421 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:49:27.055910 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:27.055882 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-bpgf8"] Jul 31 18:49:27.060542 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:27.060519 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bpgf8" Jul 31 18:49:27.065604 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:27.065578 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 31 18:49:27.065741 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:27.065697 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-rm4ld\"" Jul 31 18:49:27.092620 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:27.092529 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-bpgf8"] Jul 31 18:49:27.120663 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:27.120619 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb861fcc-08c9-427e-8a9a-bd1d9bb8c607-cert\") pod \"kserve-controller-manager-d6f4b65b8-bpgf8\" (UID: \"eb861fcc-08c9-427e-8a9a-bd1d9bb8c607\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bpgf8" Jul 31 18:49:27.120813 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:27.120743 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl695\" (UniqueName: \"kubernetes.io/projected/eb861fcc-08c9-427e-8a9a-bd1d9bb8c607-kube-api-access-vl695\") pod \"kserve-controller-manager-d6f4b65b8-bpgf8\" (UID: \"eb861fcc-08c9-427e-8a9a-bd1d9bb8c607\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bpgf8" Jul 31 18:49:27.127843 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:27.127815 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:49:27.135818 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:27.135796 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-9tbcv"] Jul 31 18:49:27.142262 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:27.142223 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9tbcv" Jul 31 18:49:27.166883 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:27.166857 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 31 18:49:27.167125 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:27.166967 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-q4fc8\"" Jul 31 18:49:27.167421 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:27.167400 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-9tbcv"] Jul 31 18:49:27.221894 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:27.221821 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb861fcc-08c9-427e-8a9a-bd1d9bb8c607-cert\") pod \"kserve-controller-manager-d6f4b65b8-bpgf8\" (UID: \"eb861fcc-08c9-427e-8a9a-bd1d9bb8c607\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bpgf8" Jul 31 18:49:27.222042 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:27.221904 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqlh4\" (UniqueName: \"kubernetes.io/projected/73c1838d-6c05-4576-8f70-b23a70a55d4f-kube-api-access-nqlh4\") pod \"llmisvc-controller-manager-777c6bd7d5-9tbcv\" (UID: \"73c1838d-6c05-4576-8f70-b23a70a55d4f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9tbcv" Jul 31 18:49:27.222042 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:27.221950 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73c1838d-6c05-4576-8f70-b23a70a55d4f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-9tbcv\" (UID: \"73c1838d-6c05-4576-8f70-b23a70a55d4f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9tbcv" Jul 31 18:49:27.222042 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:27.221992 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl695\" (UniqueName: \"kubernetes.io/projected/eb861fcc-08c9-427e-8a9a-bd1d9bb8c607-kube-api-access-vl695\") pod \"kserve-controller-manager-d6f4b65b8-bpgf8\" (UID: \"eb861fcc-08c9-427e-8a9a-bd1d9bb8c607\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bpgf8" Jul 31 18:49:27.224611 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:27.224588 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb861fcc-08c9-427e-8a9a-bd1d9bb8c607-cert\") pod \"kserve-controller-manager-d6f4b65b8-bpgf8\" (UID: \"eb861fcc-08c9-427e-8a9a-bd1d9bb8c607\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bpgf8" Jul 31 18:49:27.241003 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:27.240979 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl695\" (UniqueName: \"kubernetes.io/projected/eb861fcc-08c9-427e-8a9a-bd1d9bb8c607-kube-api-access-vl695\") pod \"kserve-controller-manager-d6f4b65b8-bpgf8\" (UID: \"eb861fcc-08c9-427e-8a9a-bd1d9bb8c607\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bpgf8" Jul 31 18:49:27.292309 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:27.292274 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:49:27.322847 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:27.322813 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqlh4\" (UniqueName: \"kubernetes.io/projected/73c1838d-6c05-4576-8f70-b23a70a55d4f-kube-api-access-nqlh4\") pod \"llmisvc-controller-manager-777c6bd7d5-9tbcv\" (UID: \"73c1838d-6c05-4576-8f70-b23a70a55d4f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9tbcv" Jul 31 18:49:27.323010 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:27.322859 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73c1838d-6c05-4576-8f70-b23a70a55d4f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-9tbcv\" (UID: \"73c1838d-6c05-4576-8f70-b23a70a55d4f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9tbcv" Jul 31 18:49:27.323088 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:49:27.323063 2565 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 31 18:49:27.323148 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:49:27.323134 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73c1838d-6c05-4576-8f70-b23a70a55d4f-cert podName:73c1838d-6c05-4576-8f70-b23a70a55d4f nodeName:}" failed. No retries permitted until 2026-07-31 18:49:27.823113454 +0000 UTC m=+344.461103233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/73c1838d-6c05-4576-8f70-b23a70a55d4f-cert") pod "llmisvc-controller-manager-777c6bd7d5-9tbcv" (UID: "73c1838d-6c05-4576-8f70-b23a70a55d4f") : secret "llmisvc-webhook-server-cert" not found Jul 31 18:49:27.339413 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:27.339387 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqlh4\" (UniqueName: \"kubernetes.io/projected/73c1838d-6c05-4576-8f70-b23a70a55d4f-kube-api-access-nqlh4\") pod \"llmisvc-controller-manager-777c6bd7d5-9tbcv\" (UID: \"73c1838d-6c05-4576-8f70-b23a70a55d4f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9tbcv" Jul 31 18:49:27.356172 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:27.356149 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" Jul 31 18:49:27.372477 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:27.372453 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bpgf8" Jul 31 18:49:27.828972 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:27.828487 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73c1838d-6c05-4576-8f70-b23a70a55d4f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-9tbcv\" (UID: \"73c1838d-6c05-4576-8f70-b23a70a55d4f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9tbcv" Jul 31 18:49:27.828972 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:49:27.828663 2565 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 31 18:49:27.828972 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:49:27.828725 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73c1838d-6c05-4576-8f70-b23a70a55d4f-cert podName:73c1838d-6c05-4576-8f70-b23a70a55d4f nodeName:}" failed. No retries permitted until 2026-07-31 18:49:28.828706071 +0000 UTC m=+345.466695866 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/73c1838d-6c05-4576-8f70-b23a70a55d4f-cert") pod "llmisvc-controller-manager-777c6bd7d5-9tbcv" (UID: "73c1838d-6c05-4576-8f70-b23a70a55d4f") : secret "llmisvc-webhook-server-cert" not found Jul 31 18:49:28.055914 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:28.055884 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:49:28.057917 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:28.057886 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 18:49:28.058021 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:28.057949 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 18:49:28.058021 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:28.057976 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 18:49:28.120705 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:28.120677 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-bpgf8"] Jul 31 18:49:28.123765 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:49:28.123725 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb861fcc_08c9_427e_8a9a_bd1d9bb8c607.slice/crio-ba68b0c4c538cb7ac106bfe4a1a9fd3c82226c899c22ea4fdaca40f0ba13e500 WatchSource:0}: Error finding container ba68b0c4c538cb7ac106bfe4a1a9fd3c82226c899c22ea4fdaca40f0ba13e500: Status 404 returned error can't find the container with id ba68b0c4c538cb7ac106bfe4a1a9fd3c82226c899c22ea4fdaca40f0ba13e500 Jul 31 18:49:28.376430 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:28.376344 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" event={"ID":"eb55cf00-4290-49f4-8367-1d2d69a90016","Type":"ContainerStarted","Data":"2df4a55fb9880cb40138bfc6d065c4581489d78f8fc095e129220ad42c4c14db"} Jul 31 18:49:28.377397 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:28.377374 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:28.377515 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:28.377412 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bpgf8" event={"ID":"eb861fcc-08c9-427e-8a9a-bd1d9bb8c607","Type":"ContainerStarted","Data":"ba68b0c4c538cb7ac106bfe4a1a9fd3c82226c899c22ea4fdaca40f0ba13e500"} Jul 31 18:49:28.390275 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:28.390224 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:49:28.438138 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:28.438097 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" podStartSLOduration=0.899385995 podStartE2EDuration="3.438085795s" totalImagesPullingTime="2.5386998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:49:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:49:28.437933798 +0000 UTC m=+345.075923602" watchObservedRunningTime="2026-07-31 18:49:28.438085795 +0000 UTC m=+345.076075594" Jul 31 18:49:28.526629 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:28.526595 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:49:28.648843 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:28.648767 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h"] Jul 31 18:49:28.648974 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:28.648954 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" podUID="e034bfa6-7b57-4146-9682-508d276a6eb2" containerName="discovery" containerID="cri-o://1953c4cd9d0b5be3b7e1e90ccd6dd9da68c526d915c8c309528204d1ae3de81b" gracePeriod=30 Jul 31 18:49:28.839926 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:28.839889 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73c1838d-6c05-4576-8f70-b23a70a55d4f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-9tbcv\" (UID: \"73c1838d-6c05-4576-8f70-b23a70a55d4f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9tbcv" Jul 31 18:49:28.842908 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:28.842882 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73c1838d-6c05-4576-8f70-b23a70a55d4f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-9tbcv\" (UID: \"73c1838d-6c05-4576-8f70-b23a70a55d4f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9tbcv" Jul 31 18:49:28.901496 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:28.901418 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" Jul 31 18:49:28.949269 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:28.941916 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e034bfa6-7b57-4146-9682-508d276a6eb2-local-certs\") pod \"e034bfa6-7b57-4146-9682-508d276a6eb2\" (UID: \"e034bfa6-7b57-4146-9682-508d276a6eb2\") " Jul 31 18:49:28.949269 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:28.941968 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e034bfa6-7b57-4146-9682-508d276a6eb2-istio-csr-dns-cert\") pod \"e034bfa6-7b57-4146-9682-508d276a6eb2\" (UID: \"e034bfa6-7b57-4146-9682-508d276a6eb2\") " Jul 31 18:49:28.949269 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:28.942034 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e034bfa6-7b57-4146-9682-508d276a6eb2-istio-csr-ca-configmap\") pod \"e034bfa6-7b57-4146-9682-508d276a6eb2\" (UID: \"e034bfa6-7b57-4146-9682-508d276a6eb2\") " Jul 31 18:49:28.949269 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:28.942078 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e034bfa6-7b57-4146-9682-508d276a6eb2-istio-kubeconfig\") pod \"e034bfa6-7b57-4146-9682-508d276a6eb2\" (UID: \"e034bfa6-7b57-4146-9682-508d276a6eb2\") " Jul 31 18:49:28.949269 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:28.942140 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qg6dw\" (UniqueName: \"kubernetes.io/projected/e034bfa6-7b57-4146-9682-508d276a6eb2-kube-api-access-qg6dw\") pod \"e034bfa6-7b57-4146-9682-508d276a6eb2\" (UID: \"e034bfa6-7b57-4146-9682-508d276a6eb2\") " Jul 31 18:49:28.949269 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:28.942176 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e034bfa6-7b57-4146-9682-508d276a6eb2-cacerts\") pod \"e034bfa6-7b57-4146-9682-508d276a6eb2\" (UID: \"e034bfa6-7b57-4146-9682-508d276a6eb2\") " Jul 31 18:49:28.949269 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:28.942229 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e034bfa6-7b57-4146-9682-508d276a6eb2-istio-token\") pod \"e034bfa6-7b57-4146-9682-508d276a6eb2\" (UID: \"e034bfa6-7b57-4146-9682-508d276a6eb2\") " Jul 31 18:49:28.952929 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:28.952738 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e034bfa6-7b57-4146-9682-508d276a6eb2-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "e034bfa6-7b57-4146-9682-508d276a6eb2" (UID: "e034bfa6-7b57-4146-9682-508d276a6eb2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 18:49:28.952929 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:28.952794 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9tbcv" Jul 31 18:49:28.953602 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:28.953573 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e034bfa6-7b57-4146-9682-508d276a6eb2-istio-token" (OuterVolumeSpecName: "istio-token") pod "e034bfa6-7b57-4146-9682-508d276a6eb2" (UID: "e034bfa6-7b57-4146-9682-508d276a6eb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 18:49:28.955137 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:28.955110 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e034bfa6-7b57-4146-9682-508d276a6eb2-local-certs" (OuterVolumeSpecName: "local-certs") pod "e034bfa6-7b57-4146-9682-508d276a6eb2" (UID: "e034bfa6-7b57-4146-9682-508d276a6eb2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 18:49:28.955757 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:28.955727 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e034bfa6-7b57-4146-9682-508d276a6eb2-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "e034bfa6-7b57-4146-9682-508d276a6eb2" (UID: "e034bfa6-7b57-4146-9682-508d276a6eb2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:49:28.956302 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:28.956269 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e034bfa6-7b57-4146-9682-508d276a6eb2-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "e034bfa6-7b57-4146-9682-508d276a6eb2" (UID: "e034bfa6-7b57-4146-9682-508d276a6eb2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:49:28.956603 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:28.956583 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e034bfa6-7b57-4146-9682-508d276a6eb2-cacerts" (OuterVolumeSpecName: "cacerts") pod "e034bfa6-7b57-4146-9682-508d276a6eb2" (UID: "e034bfa6-7b57-4146-9682-508d276a6eb2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:49:28.957401 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:28.957368 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e034bfa6-7b57-4146-9682-508d276a6eb2-kube-api-access-qg6dw" (OuterVolumeSpecName: "kube-api-access-qg6dw") pod "e034bfa6-7b57-4146-9682-508d276a6eb2" (UID: "e034bfa6-7b57-4146-9682-508d276a6eb2"). InnerVolumeSpecName "kube-api-access-qg6dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 18:49:29.043305 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:29.043266 2565 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e034bfa6-7b57-4146-9682-508d276a6eb2-cacerts\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:49:29.043305 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:29.043300 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e034bfa6-7b57-4146-9682-508d276a6eb2-istio-token\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:49:29.043513 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:29.043315 2565 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e034bfa6-7b57-4146-9682-508d276a6eb2-local-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:49:29.043513 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:29.043327 2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e034bfa6-7b57-4146-9682-508d276a6eb2-istio-csr-dns-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:49:29.043513 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:29.043340 2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e034bfa6-7b57-4146-9682-508d276a6eb2-istio-csr-ca-configmap\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:49:29.043513 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:29.043352 2565 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e034bfa6-7b57-4146-9682-508d276a6eb2-istio-kubeconfig\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:49:29.043513 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:29.043363 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg6dw\" (UniqueName: \"kubernetes.io/projected/e034bfa6-7b57-4146-9682-508d276a6eb2-kube-api-access-qg6dw\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:49:29.112045 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:29.112016 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-9tbcv"] Jul 31 18:49:29.114151 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:49:29.114122 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod73c1838d_6c05_4576_8f70_b23a70a55d4f.slice/crio-f1a457433f87af2c42b2827aa3d174c0b79627abd0f6e50700e79d2f1098aac2 WatchSource:0}: Error finding container f1a457433f87af2c42b2827aa3d174c0b79627abd0f6e50700e79d2f1098aac2: Status 404 returned error can't find the container with id f1a457433f87af2c42b2827aa3d174c0b79627abd0f6e50700e79d2f1098aac2 Jul 31 18:49:29.382812 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:29.382783 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:29.383132 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:29.383105 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9tbcv" event={"ID":"73c1838d-6c05-4576-8f70-b23a70a55d4f","Type":"ContainerStarted","Data":"f1a457433f87af2c42b2827aa3d174c0b79627abd0f6e50700e79d2f1098aac2"} Jul 31 18:49:29.384283 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:29.384260 2565 generic.go:356] "Generic (PLEG): container finished" podID="e034bfa6-7b57-4146-9682-508d276a6eb2" containerID="1953c4cd9d0b5be3b7e1e90ccd6dd9da68c526d915c8c309528204d1ae3de81b" exitCode=0 Jul 31 18:49:29.384387 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:29.384352 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" Jul 31 18:49:29.384446 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:29.384398 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" event={"ID":"e034bfa6-7b57-4146-9682-508d276a6eb2","Type":"ContainerDied","Data":"1953c4cd9d0b5be3b7e1e90ccd6dd9da68c526d915c8c309528204d1ae3de81b"} Jul 31 18:49:29.384507 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:29.384445 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h" event={"ID":"e034bfa6-7b57-4146-9682-508d276a6eb2","Type":"ContainerDied","Data":"baa1e2ce9ce536bb12a5e512e505817fcc33bac952bafba3b8801066cabda752"} Jul 31 18:49:29.384507 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:29.384474 2565 scope.go:117] "RemoveContainer" containerID="1953c4cd9d0b5be3b7e1e90ccd6dd9da68c526d915c8c309528204d1ae3de81b" Jul 31 18:49:29.384634 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:29.384614 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:29.385537 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:29.385517 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d" Jul 31 18:49:29.393413 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:29.393393 2565 scope.go:117] "RemoveContainer" containerID="1953c4cd9d0b5be3b7e1e90ccd6dd9da68c526d915c8c309528204d1ae3de81b" Jul 31 18:49:29.393655 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:49:29.393636 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1953c4cd9d0b5be3b7e1e90ccd6dd9da68c526d915c8c309528204d1ae3de81b\": container with ID starting with 1953c4cd9d0b5be3b7e1e90ccd6dd9da68c526d915c8c309528204d1ae3de81b not found: ID does not exist" containerID="1953c4cd9d0b5be3b7e1e90ccd6dd9da68c526d915c8c309528204d1ae3de81b" Jul 31 18:49:29.393713 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:29.393664 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1953c4cd9d0b5be3b7e1e90ccd6dd9da68c526d915c8c309528204d1ae3de81b"} err="failed to get container status \"1953c4cd9d0b5be3b7e1e90ccd6dd9da68c526d915c8c309528204d1ae3de81b\": rpc error: code = NotFound desc = could not find container \"1953c4cd9d0b5be3b7e1e90ccd6dd9da68c526d915c8c309528204d1ae3de81b\": container with ID starting with 1953c4cd9d0b5be3b7e1e90ccd6dd9da68c526d915c8c309528204d1ae3de81b not found: ID does not exist" Jul 31 18:49:29.446686 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:29.446640 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h"] Jul 31 18:49:29.453926 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:29.453893 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vgs4h"] Jul 31 18:49:29.888403 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:29.888367 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e034bfa6-7b57-4146-9682-508d276a6eb2" path="/var/lib/kubelet/pods/e034bfa6-7b57-4146-9682-508d276a6eb2/volumes" Jul 31 18:49:31.399748 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:31.399662 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bpgf8" event={"ID":"eb861fcc-08c9-427e-8a9a-bd1d9bb8c607","Type":"ContainerStarted","Data":"5cf96c17e1d06e777868b68f82d3ff6213bb417b60dea0781eb318a55ad195a8"} Jul 31 18:49:31.400086 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:31.399813 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bpgf8" Jul 31 18:49:31.419514 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:31.419456 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bpgf8" podStartSLOduration=1.782614464 podStartE2EDuration="4.419443084s" totalImagesPullingTime="2.63682862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:49:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:49:31.417737968 +0000 UTC m=+348.055727809" watchObservedRunningTime="2026-07-31 18:49:31.419443084 +0000 UTC m=+348.057432929" Jul 31 18:49:34.357523 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:34.357492 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-ktlkc" Jul 31 18:49:34.359919 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:34.359895 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-wnvkm" Jul 31 18:49:34.414955 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:34.414912 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9tbcv" event={"ID":"73c1838d-6c05-4576-8f70-b23a70a55d4f","Type":"ContainerStarted","Data":"1f5b60a247586291a3d009a2de4dfddfb488842950785b35577bee8b1901a250"} Jul 31 18:49:34.415186 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:34.415162 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9tbcv" Jul 31 18:49:34.443021 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:34.442968 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9tbcv" podStartSLOduration=2.875082961 podStartE2EDuration="7.442952419s" totalImagesPullingTime="4.567869458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:49:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:49:34.44126811 +0000 UTC m=+351.079257910" watchObservedRunningTime="2026-07-31 18:49:34.442952419 +0000 UTC m=+351.080942220" Jul 31 18:49:41.269988 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:41.269896 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:49:56.739156 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:56.739116 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:49:56.803766 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:49:56.803734 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:50:02.409508 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:02.409476 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bpgf8" Jul 31 18:50:05.421827 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:05.421796 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9tbcv" Jul 31 18:50:17.635973 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:17.635934 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:50:17.736263 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:17.736212 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:50:30.848474 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:30.848436 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:50:33.101113 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:33.101076 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fczlj"] Jul 31 18:50:33.101490 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:33.101429 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e034bfa6-7b57-4146-9682-508d276a6eb2" containerName="discovery" Jul 31 18:50:33.101490 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:33.101443 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e034bfa6-7b57-4146-9682-508d276a6eb2" containerName="discovery" Jul 31 18:50:33.101561 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:33.101504 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e034bfa6-7b57-4146-9682-508d276a6eb2" containerName="discovery" Jul 31 18:50:33.103664 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:33.103648 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fczlj" Jul 31 18:50:33.106047 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:33.106027 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 31 18:50:33.106552 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:33.106535 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 31 18:50:33.106844 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:33.106829 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-9gswt\"" Jul 31 18:50:33.119216 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:33.119195 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fczlj"] Jul 31 18:50:33.170107 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:33.170084 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg8pz\" (UniqueName: \"kubernetes.io/projected/9d3a24b1-2a71-4af7-92f6-6af1eb2c2d01-kube-api-access-tg8pz\") pod \"limitador-operator-controller-manager-6c549b6d85-fczlj\" (UID: \"9d3a24b1-2a71-4af7-92f6-6af1eb2c2d01\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fczlj" Jul 31 18:50:33.271496 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:33.271458 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg8pz\" (UniqueName: \"kubernetes.io/projected/9d3a24b1-2a71-4af7-92f6-6af1eb2c2d01-kube-api-access-tg8pz\") pod \"limitador-operator-controller-manager-6c549b6d85-fczlj\" (UID: \"9d3a24b1-2a71-4af7-92f6-6af1eb2c2d01\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fczlj" Jul 31 18:50:33.289102 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:33.289072 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg8pz\" (UniqueName: \"kubernetes.io/projected/9d3a24b1-2a71-4af7-92f6-6af1eb2c2d01-kube-api-access-tg8pz\") pod \"limitador-operator-controller-manager-6c549b6d85-fczlj\" (UID: \"9d3a24b1-2a71-4af7-92f6-6af1eb2c2d01\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fczlj" Jul 31 18:50:33.413738 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:33.413660 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fczlj" Jul 31 18:50:33.546547 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:33.546523 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fczlj"] Jul 31 18:50:33.548398 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:50:33.548359 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d3a24b1_2a71_4af7_92f6_6af1eb2c2d01.slice/crio-a839fd375e9242b73be7d26f340e69e24195845bf924a2ca86ae8fda1a4e251d WatchSource:0}: Error finding container a839fd375e9242b73be7d26f340e69e24195845bf924a2ca86ae8fda1a4e251d: Status 404 returned error can't find the container with id a839fd375e9242b73be7d26f340e69e24195845bf924a2ca86ae8fda1a4e251d Jul 31 18:50:33.647044 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:33.647011 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:50:33.669713 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:33.669649 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fczlj" event={"ID":"9d3a24b1-2a71-4af7-92f6-6af1eb2c2d01","Type":"ContainerStarted","Data":"a839fd375e9242b73be7d26f340e69e24195845bf924a2ca86ae8fda1a4e251d"} Jul 31 18:50:35.154306 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.154270 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-5k7rm"] Jul 31 18:50:35.156786 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.156763 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-5k7rm" Jul 31 18:50:35.160558 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.160520 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-zk29d\"" Jul 31 18:50:35.160961 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.160692 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 31 18:50:35.180146 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.180123 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-5k7rm"] Jul 31 18:50:35.189026 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.188985 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-772rc\" (UniqueName: \"kubernetes.io/projected/d4fcf557-31a9-4bd9-acdd-6b12028632f2-kube-api-access-772rc\") pod \"dns-operator-controller-manager-5985564776-5k7rm\" (UID: \"d4fcf557-31a9-4bd9-acdd-6b12028632f2\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-5k7rm" Jul 31 18:50:35.263066 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.263031 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:50:35.290250 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.290201 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-772rc\" (UniqueName: \"kubernetes.io/projected/d4fcf557-31a9-4bd9-acdd-6b12028632f2-kube-api-access-772rc\") pod \"dns-operator-controller-manager-5985564776-5k7rm\" (UID: \"d4fcf557-31a9-4bd9-acdd-6b12028632f2\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-5k7rm" Jul 31 18:50:35.293171 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.293144 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c8df6ddb9-mzdgl"] Jul 31 18:50:35.297685 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.297668 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c8df6ddb9-mzdgl" Jul 31 18:50:35.316140 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.316086 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-772rc\" (UniqueName: \"kubernetes.io/projected/d4fcf557-31a9-4bd9-acdd-6b12028632f2-kube-api-access-772rc\") pod \"dns-operator-controller-manager-5985564776-5k7rm\" (UID: \"d4fcf557-31a9-4bd9-acdd-6b12028632f2\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-5k7rm" Jul 31 18:50:35.316276 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.316153 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c8df6ddb9-mzdgl"] Jul 31 18:50:35.390775 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.390745 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e5b8c507-5033-476e-8c9f-dd129e98e2b0-oauth-serving-cert\") pod \"console-5c8df6ddb9-mzdgl\" (UID: \"e5b8c507-5033-476e-8c9f-dd129e98e2b0\") " pod="openshift-console/console-5c8df6ddb9-mzdgl" Jul 31 18:50:35.390775 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.390768 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:50:35.391002 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.390795 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e5b8c507-5033-476e-8c9f-dd129e98e2b0-console-serving-cert\") pod \"console-5c8df6ddb9-mzdgl\" (UID: \"e5b8c507-5033-476e-8c9f-dd129e98e2b0\") " pod="openshift-console/console-5c8df6ddb9-mzdgl" Jul 31 18:50:35.391002 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.390940 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e5b8c507-5033-476e-8c9f-dd129e98e2b0-console-config\") pod \"console-5c8df6ddb9-mzdgl\" (UID: \"e5b8c507-5033-476e-8c9f-dd129e98e2b0\") " pod="openshift-console/console-5c8df6ddb9-mzdgl" Jul 31 18:50:35.391098 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.391021 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5b8c507-5033-476e-8c9f-dd129e98e2b0-trusted-ca-bundle\") pod \"console-5c8df6ddb9-mzdgl\" (UID: \"e5b8c507-5033-476e-8c9f-dd129e98e2b0\") " pod="openshift-console/console-5c8df6ddb9-mzdgl" Jul 31 18:50:35.391098 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.391044 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e5b8c507-5033-476e-8c9f-dd129e98e2b0-console-oauth-config\") pod \"console-5c8df6ddb9-mzdgl\" (UID: \"e5b8c507-5033-476e-8c9f-dd129e98e2b0\") " pod="openshift-console/console-5c8df6ddb9-mzdgl" Jul 31 18:50:35.391098 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.391062 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e5b8c507-5033-476e-8c9f-dd129e98e2b0-service-ca\") pod \"console-5c8df6ddb9-mzdgl\" (UID: \"e5b8c507-5033-476e-8c9f-dd129e98e2b0\") " pod="openshift-console/console-5c8df6ddb9-mzdgl" Jul 31 18:50:35.391098 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.391085 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9drl\" (UniqueName: \"kubernetes.io/projected/e5b8c507-5033-476e-8c9f-dd129e98e2b0-kube-api-access-m9drl\") pod \"console-5c8df6ddb9-mzdgl\" (UID: \"e5b8c507-5033-476e-8c9f-dd129e98e2b0\") " pod="openshift-console/console-5c8df6ddb9-mzdgl" Jul 31 18:50:35.469436 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.469359 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-5k7rm" Jul 31 18:50:35.491945 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.491911 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e5b8c507-5033-476e-8c9f-dd129e98e2b0-console-config\") pod \"console-5c8df6ddb9-mzdgl\" (UID: \"e5b8c507-5033-476e-8c9f-dd129e98e2b0\") " pod="openshift-console/console-5c8df6ddb9-mzdgl" Jul 31 18:50:35.492073 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.491984 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5b8c507-5033-476e-8c9f-dd129e98e2b0-trusted-ca-bundle\") pod \"console-5c8df6ddb9-mzdgl\" (UID: \"e5b8c507-5033-476e-8c9f-dd129e98e2b0\") " pod="openshift-console/console-5c8df6ddb9-mzdgl" Jul 31 18:50:35.492073 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.492013 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e5b8c507-5033-476e-8c9f-dd129e98e2b0-console-oauth-config\") pod \"console-5c8df6ddb9-mzdgl\" (UID: \"e5b8c507-5033-476e-8c9f-dd129e98e2b0\") " pod="openshift-console/console-5c8df6ddb9-mzdgl" Jul 31 18:50:35.492073 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.492039 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e5b8c507-5033-476e-8c9f-dd129e98e2b0-service-ca\") pod \"console-5c8df6ddb9-mzdgl\" (UID: \"e5b8c507-5033-476e-8c9f-dd129e98e2b0\") " pod="openshift-console/console-5c8df6ddb9-mzdgl" Jul 31 18:50:35.492073 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.492062 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9drl\" (UniqueName: \"kubernetes.io/projected/e5b8c507-5033-476e-8c9f-dd129e98e2b0-kube-api-access-m9drl\") pod \"console-5c8df6ddb9-mzdgl\" (UID: \"e5b8c507-5033-476e-8c9f-dd129e98e2b0\") " pod="openshift-console/console-5c8df6ddb9-mzdgl" Jul 31 18:50:35.492340 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.492114 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e5b8c507-5033-476e-8c9f-dd129e98e2b0-oauth-serving-cert\") pod \"console-5c8df6ddb9-mzdgl\" (UID: \"e5b8c507-5033-476e-8c9f-dd129e98e2b0\") " pod="openshift-console/console-5c8df6ddb9-mzdgl" Jul 31 18:50:35.493565 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.493083 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e5b8c507-5033-476e-8c9f-dd129e98e2b0-console-config\") pod \"console-5c8df6ddb9-mzdgl\" (UID: \"e5b8c507-5033-476e-8c9f-dd129e98e2b0\") " pod="openshift-console/console-5c8df6ddb9-mzdgl" Jul 31 18:50:35.493565 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.493182 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e5b8c507-5033-476e-8c9f-dd129e98e2b0-console-serving-cert\") pod \"console-5c8df6ddb9-mzdgl\" (UID: \"e5b8c507-5033-476e-8c9f-dd129e98e2b0\") " pod="openshift-console/console-5c8df6ddb9-mzdgl" Jul 31 18:50:35.493565 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.493508 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5b8c507-5033-476e-8c9f-dd129e98e2b0-trusted-ca-bundle\") pod \"console-5c8df6ddb9-mzdgl\" (UID: \"e5b8c507-5033-476e-8c9f-dd129e98e2b0\") " pod="openshift-console/console-5c8df6ddb9-mzdgl" Jul 31 18:50:35.497870 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.493876 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e5b8c507-5033-476e-8c9f-dd129e98e2b0-service-ca\") pod \"console-5c8df6ddb9-mzdgl\" (UID: \"e5b8c507-5033-476e-8c9f-dd129e98e2b0\") " pod="openshift-console/console-5c8df6ddb9-mzdgl" Jul 31 18:50:35.497870 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.494212 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e5b8c507-5033-476e-8c9f-dd129e98e2b0-oauth-serving-cert\") pod \"console-5c8df6ddb9-mzdgl\" (UID: \"e5b8c507-5033-476e-8c9f-dd129e98e2b0\") " pod="openshift-console/console-5c8df6ddb9-mzdgl" Jul 31 18:50:35.497870 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.495314 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:50:35.500844 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.500822 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e5b8c507-5033-476e-8c9f-dd129e98e2b0-console-serving-cert\") pod \"console-5c8df6ddb9-mzdgl\" (UID: \"e5b8c507-5033-476e-8c9f-dd129e98e2b0\") " pod="openshift-console/console-5c8df6ddb9-mzdgl" Jul 31 18:50:35.500943 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.500846 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e5b8c507-5033-476e-8c9f-dd129e98e2b0-console-oauth-config\") pod \"console-5c8df6ddb9-mzdgl\" (UID: \"e5b8c507-5033-476e-8c9f-dd129e98e2b0\") " pod="openshift-console/console-5c8df6ddb9-mzdgl" Jul 31 18:50:35.504917 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.504898 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9drl\" (UniqueName: \"kubernetes.io/projected/e5b8c507-5033-476e-8c9f-dd129e98e2b0-kube-api-access-m9drl\") pod \"console-5c8df6ddb9-mzdgl\" (UID: \"e5b8c507-5033-476e-8c9f-dd129e98e2b0\") " pod="openshift-console/console-5c8df6ddb9-mzdgl" Jul 31 18:50:35.608754 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.608718 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c8df6ddb9-mzdgl" Jul 31 18:50:35.629889 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.629803 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:50:35.746619 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.746583 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:50:35.876250 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:35.876197 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:50:36.312315 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:36.312277 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c8df6ddb9-mzdgl"] Jul 31 18:50:36.327955 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:50:36.327926 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5b8c507_5033_476e_8c9f_dd129e98e2b0.slice/crio-bc21a49ab517f26f537fdf079705c9e932a42cdb5f47d3b3bf0b5952f6ba7212 WatchSource:0}: Error finding container bc21a49ab517f26f537fdf079705c9e932a42cdb5f47d3b3bf0b5952f6ba7212: Status 404 returned error can't find the container with id bc21a49ab517f26f537fdf079705c9e932a42cdb5f47d3b3bf0b5952f6ba7212 Jul 31 18:50:36.345106 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:36.343157 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-5k7rm"] Jul 31 18:50:36.347001 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:50:36.346958 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4fcf557_31a9_4bd9_acdd_6b12028632f2.slice/crio-3b28f06d870d5a4a65ba56c54a165a2d3cb2e2d6011c8a1b3a1521efdce3bb7a WatchSource:0}: Error finding container 3b28f06d870d5a4a65ba56c54a165a2d3cb2e2d6011c8a1b3a1521efdce3bb7a: Status 404 returned error can't find the container with id 3b28f06d870d5a4a65ba56c54a165a2d3cb2e2d6011c8a1b3a1521efdce3bb7a Jul 31 18:50:36.687433 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:36.687397 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fczlj" event={"ID":"9d3a24b1-2a71-4af7-92f6-6af1eb2c2d01","Type":"ContainerStarted","Data":"e2a63784d5a5c4fd92b7a3fccb98a1e2c77908dc0658e82d410919fd9c2ad73d"} Jul 31 18:50:36.687634 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:36.687498 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fczlj" Jul 31 18:50:36.688744 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:36.688719 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-5k7rm" event={"ID":"d4fcf557-31a9-4bd9-acdd-6b12028632f2","Type":"ContainerStarted","Data":"3b28f06d870d5a4a65ba56c54a165a2d3cb2e2d6011c8a1b3a1521efdce3bb7a"} Jul 31 18:50:36.690040 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:36.690016 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c8df6ddb9-mzdgl" event={"ID":"e5b8c507-5033-476e-8c9f-dd129e98e2b0","Type":"ContainerStarted","Data":"e6ad73cfbec0644651baaed0de2ae7a563feb92ca098fd85940ee29dc764f40f"} Jul 31 18:50:36.690127 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:36.690055 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c8df6ddb9-mzdgl" event={"ID":"e5b8c507-5033-476e-8c9f-dd129e98e2b0","Type":"ContainerStarted","Data":"bc21a49ab517f26f537fdf079705c9e932a42cdb5f47d3b3bf0b5952f6ba7212"} Jul 31 18:50:36.726000 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:36.725958 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fczlj" podStartSLOduration=1.079956618 podStartE2EDuration="3.725943538s" totalImagesPullingTime="2.64598692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:50:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:50:36.724461668 +0000 UTC m=+413.362451481" watchObservedRunningTime="2026-07-31 18:50:36.725943538 +0000 UTC m=+413.363933337" Jul 31 18:50:36.759162 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:36.759115 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c8df6ddb9-mzdgl" podStartSLOduration=1.7590968089999999 podStartE2EDuration="1.759096809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:50:36.757905533 +0000 UTC m=+413.395895334" watchObservedRunningTime="2026-07-31 18:50:36.759096809 +0000 UTC m=+413.397086610" Jul 31 18:50:36.907034 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:36.906998 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:50:37.925133 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:37.925109 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-4z7w6"] Jul 31 18:50:37.927854 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:37.927839 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-4z7w6" Jul 31 18:50:37.931761 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:37.931743 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-mx9dd\"" Jul 31 18:50:37.967673 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:37.967650 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-4z7w6"] Jul 31 18:50:38.018337 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:38.018293 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z42nm\" (UniqueName: \"kubernetes.io/projected/fb30c671-c6c4-4301-9a08-0a72025c1fc5-kube-api-access-z42nm\") pod \"authorino-operator-68d88948d8-4z7w6\" (UID: \"fb30c671-c6c4-4301-9a08-0a72025c1fc5\") " pod="kuadrant-system/authorino-operator-68d88948d8-4z7w6" Jul 31 18:50:38.118930 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:38.118897 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z42nm\" (UniqueName: \"kubernetes.io/projected/fb30c671-c6c4-4301-9a08-0a72025c1fc5-kube-api-access-z42nm\") pod \"authorino-operator-68d88948d8-4z7w6\" (UID: \"fb30c671-c6c4-4301-9a08-0a72025c1fc5\") " pod="kuadrant-system/authorino-operator-68d88948d8-4z7w6" Jul 31 18:50:38.149798 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:38.149769 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z42nm\" (UniqueName: \"kubernetes.io/projected/fb30c671-c6c4-4301-9a08-0a72025c1fc5-kube-api-access-z42nm\") pod \"authorino-operator-68d88948d8-4z7w6\" (UID: \"fb30c671-c6c4-4301-9a08-0a72025c1fc5\") " pod="kuadrant-system/authorino-operator-68d88948d8-4z7w6" Jul 31 18:50:38.296974 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:38.296946 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-4z7w6" Jul 31 18:50:38.354073 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:38.354032 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:50:38.443893 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:38.443861 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-4z7w6"] Jul 31 18:50:38.447138 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:50:38.447117 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb30c671_c6c4_4301_9a08_0a72025c1fc5.slice/crio-5689fa7cbb025d3a8f62843fdadb82b558487dc6d7b2493e99472e5a20e281ef WatchSource:0}: Error finding container 5689fa7cbb025d3a8f62843fdadb82b558487dc6d7b2493e99472e5a20e281ef: Status 404 returned error can't find the container with id 5689fa7cbb025d3a8f62843fdadb82b558487dc6d7b2493e99472e5a20e281ef Jul 31 18:50:38.702494 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:38.702401 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-5k7rm" event={"ID":"d4fcf557-31a9-4bd9-acdd-6b12028632f2","Type":"ContainerStarted","Data":"0dd51e2cd087b5be70fe757b74114710e5722927cb1556939f9bd73e9fca67ef"} Jul 31 18:50:38.702666 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:38.702553 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-5k7rm" Jul 31 18:50:38.703661 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:38.703639 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-4z7w6" event={"ID":"fb30c671-c6c4-4301-9a08-0a72025c1fc5","Type":"ContainerStarted","Data":"5689fa7cbb025d3a8f62843fdadb82b558487dc6d7b2493e99472e5a20e281ef"} Jul 31 18:50:38.733636 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:38.733587 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5985564776-5k7rm" podStartSLOduration=2.173775467 podStartE2EDuration="3.733572246s" totalImagesPullingTime="1.559796779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:50:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:50:38.732495785 +0000 UTC m=+415.370485585" watchObservedRunningTime="2026-07-31 18:50:38.733572246 +0000 UTC m=+415.371562045" Jul 31 18:50:38.892182 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:38.892148 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:50:39.046006 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:39.045973 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:50:39.341304 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:39.341204 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:50:39.469154 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:39.469092 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:50:39.705376 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:39.705144 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:50:39.873983 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:39.873935 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:50:40.717398 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:40.717362 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-4z7w6" event={"ID":"fb30c671-c6c4-4301-9a08-0a72025c1fc5","Type":"ContainerStarted","Data":"b28700935845eee0b5063bfa6693077d8ce6acfcd3a3e14f50dd703ebb440fdc"} Jul 31 18:50:40.717790 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:40.717494 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d88948d8-4z7w6" Jul 31 18:50:40.780100 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:40.779850 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d88948d8-4z7w6" podStartSLOduration=2.3132093559999998 podStartE2EDuration="3.779830095s" totalImagesPullingTime="1.466620739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:50:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:50:40.779457121 +0000 UTC m=+417.417446926" watchObservedRunningTime="2026-07-31 18:50:40.779830095 +0000 UTC m=+417.417819896" Jul 31 18:50:44.669555 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:44.669521 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:50:44.815175 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:44.815143 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:50:45.609864 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:45.609830 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c8df6ddb9-mzdgl" Jul 31 18:50:45.610037 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:45.609914 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c8df6ddb9-mzdgl" Jul 31 18:50:45.614630 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:45.614608 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c8df6ddb9-mzdgl" Jul 31 18:50:45.741603 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:45.741577 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c8df6ddb9-mzdgl" Jul 31 18:50:45.832137 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:45.832102 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5648497788-ctvkm"] Jul 31 18:50:46.087771 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:46.087736 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:50:46.260484 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:46.260434 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:50:46.494053 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:46.493941 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:50:47.698320 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:47.698290 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fczlj" Jul 31 18:50:49.714393 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:49.714363 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-5k7rm" Jul 31 18:50:51.438463 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:51.438426 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-9fgbg"] Jul 31 18:50:51.441027 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:51.441008 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-9fgbg" Jul 31 18:50:51.442708 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:51.442679 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:50:51.443529 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:51.443504 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 31 18:50:51.443614 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:51.443565 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 31 18:50:51.443666 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:51.443623 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-57xp7\"" Jul 31 18:50:51.456428 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:51.456399 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-9fgbg"] Jul 31 18:50:51.532933 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:51.532897 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:50:51.539962 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:51.539939 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7398c822-ac51-47b6-acef-cb622d693145-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-9fgbg\" (UID: \"7398c822-ac51-47b6-acef-cb622d693145\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-9fgbg" Jul 31 18:50:51.540051 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:51.539966 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7398c822-ac51-47b6-acef-cb622d693145-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-9fgbg\" (UID: \"7398c822-ac51-47b6-acef-cb622d693145\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-9fgbg" Jul 31 18:50:51.540051 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:51.540008 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbbhl\" (UniqueName: \"kubernetes.io/projected/7398c822-ac51-47b6-acef-cb622d693145-kube-api-access-dbbhl\") pod \"kuadrant-console-plugin-598f86468b-9fgbg\" (UID: \"7398c822-ac51-47b6-acef-cb622d693145\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-9fgbg" Jul 31 18:50:51.641293 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:51.641231 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7398c822-ac51-47b6-acef-cb622d693145-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-9fgbg\" (UID: \"7398c822-ac51-47b6-acef-cb622d693145\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-9fgbg" Jul 31 18:50:51.641491 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:51.641293 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7398c822-ac51-47b6-acef-cb622d693145-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-9fgbg\" (UID: \"7398c822-ac51-47b6-acef-cb622d693145\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-9fgbg" Jul 31 18:50:51.641491 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:51.641378 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbbhl\" (UniqueName: \"kubernetes.io/projected/7398c822-ac51-47b6-acef-cb622d693145-kube-api-access-dbbhl\") pod \"kuadrant-console-plugin-598f86468b-9fgbg\" (UID: \"7398c822-ac51-47b6-acef-cb622d693145\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-9fgbg" Jul 31 18:50:51.641988 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:51.641967 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7398c822-ac51-47b6-acef-cb622d693145-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-9fgbg\" (UID: \"7398c822-ac51-47b6-acef-cb622d693145\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-9fgbg" Jul 31 18:50:51.643879 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:51.643846 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7398c822-ac51-47b6-acef-cb622d693145-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-9fgbg\" (UID: \"7398c822-ac51-47b6-acef-cb622d693145\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-9fgbg" Jul 31 18:50:51.652621 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:51.652598 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbbhl\" (UniqueName: \"kubernetes.io/projected/7398c822-ac51-47b6-acef-cb622d693145-kube-api-access-dbbhl\") pod \"kuadrant-console-plugin-598f86468b-9fgbg\" (UID: \"7398c822-ac51-47b6-acef-cb622d693145\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-9fgbg" Jul 31 18:50:51.723840 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:51.723757 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d88948d8-4z7w6" Jul 31 18:50:51.750605 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:51.750571 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-9fgbg" Jul 31 18:50:51.903188 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:51.903155 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-9fgbg"] Jul 31 18:50:51.906266 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:50:51.906196 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7398c822_ac51_47b6_acef_cb622d693145.slice/crio-81ae7a143b4c666721148c73fac7f35b3bfe6067c6a02277ed1dedf6db7676a5 WatchSource:0}: Error finding container 81ae7a143b4c666721148c73fac7f35b3bfe6067c6a02277ed1dedf6db7676a5: Status 404 returned error can't find the container with id 81ae7a143b4c666721148c73fac7f35b3bfe6067c6a02277ed1dedf6db7676a5 Jul 31 18:50:52.584623 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:52.584589 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:50:52.714659 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:52.714619 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:50:52.771833 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:52.771802 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-9fgbg" event={"ID":"7398c822-ac51-47b6-acef-cb622d693145","Type":"ContainerStarted","Data":"81ae7a143b4c666721148c73fac7f35b3bfe6067c6a02277ed1dedf6db7676a5"} Jul 31 18:50:56.793742 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:56.793706 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-9fgbg" event={"ID":"7398c822-ac51-47b6-acef-cb622d693145","Type":"ContainerStarted","Data":"e0e5e741d0dc46380c5c048e915d18c213fa015ee3de5a9df96553a46b4cf85c"} Jul 31 18:50:56.812170 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:56.812123 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-9fgbg" podStartSLOduration=1.066625172 podStartE2EDuration="5.812109852s" totalImagesPullingTime="4.74548468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:50:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:50:56.81020485 +0000 UTC m=+433.448194649" watchObservedRunningTime="2026-07-31 18:50:56.812109852 +0000 UTC m=+433.450099651" Jul 31 18:50:57.207604 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:57.207567 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:50:57.308450 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:57.308420 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:50:57.409427 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:57.409397 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:50:57.878990 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:50:57.878954 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:51:01.146969 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.146932 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-4djvz"] Jul 31 18:51:01.149722 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.149697 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-4djvz" Jul 31 18:51:01.152693 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.152669 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5kvcw\"" Jul 31 18:51:01.173149 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.173119 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-4djvz"] Jul 31 18:51:01.327077 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.327043 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70aa0976-4b52-4070-8790-d799f5871e2c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-4djvz\" (UID: \"70aa0976-4b52-4070-8790-d799f5871e2c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-4djvz" Jul 31 18:51:01.327221 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.327101 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmnn4\" (UniqueName: \"kubernetes.io/projected/70aa0976-4b52-4070-8790-d799f5871e2c-kube-api-access-kmnn4\") pod \"kuadrant-operator-controller-manager-6bd956ccf-4djvz\" (UID: \"70aa0976-4b52-4070-8790-d799f5871e2c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-4djvz" Jul 31 18:51:01.427671 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.427578 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70aa0976-4b52-4070-8790-d799f5871e2c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-4djvz\" (UID: \"70aa0976-4b52-4070-8790-d799f5871e2c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-4djvz" Jul 31 18:51:01.427671 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.427640 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmnn4\" (UniqueName: \"kubernetes.io/projected/70aa0976-4b52-4070-8790-d799f5871e2c-kube-api-access-kmnn4\") pod \"kuadrant-operator-controller-manager-6bd956ccf-4djvz\" (UID: \"70aa0976-4b52-4070-8790-d799f5871e2c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-4djvz" Jul 31 18:51:01.427978 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.427957 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70aa0976-4b52-4070-8790-d799f5871e2c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-4djvz\" (UID: \"70aa0976-4b52-4070-8790-d799f5871e2c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-4djvz" Jul 31 18:51:01.437318 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.437288 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmnn4\" (UniqueName: \"kubernetes.io/projected/70aa0976-4b52-4070-8790-d799f5871e2c-kube-api-access-kmnn4\") pod \"kuadrant-operator-controller-manager-6bd956ccf-4djvz\" (UID: \"70aa0976-4b52-4070-8790-d799f5871e2c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-4djvz" Jul 31 18:51:01.459120 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.459081 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-4djvz" Jul 31 18:51:01.515195 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.512054 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-4djvz"] Jul 31 18:51:01.515833 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.515807 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-4djvz"] Jul 31 18:51:01.527327 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.526781 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fczlj"] Jul 31 18:51:01.527327 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.527023 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fczlj" podUID="9d3a24b1-2a71-4af7-92f6-6af1eb2c2d01" containerName="manager" containerID="cri-o://e2a63784d5a5c4fd92b7a3fccb98a1e2c77908dc0658e82d410919fd9c2ad73d" gracePeriod=2 Jul 31 18:51:01.543904 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.542860 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-fpbkf"] Jul 31 18:51:01.546035 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.546014 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-fpbkf" Jul 31 18:51:01.560395 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.560362 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fczlj"] Jul 31 18:51:01.577032 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:51:01.576996 2565 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"9d3a24b1-2a71-4af7-92f6-6af1eb2c2d01\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-6c549b6d85-fczlj\": pods \"limitador-operator-controller-manager-6c549b6d85-fczlj\" not found" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fczlj" Jul 31 18:51:01.598730 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.598701 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-fpbkf"] Jul 31 18:51:01.628848 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.628811 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8gs4\" (UniqueName: \"kubernetes.io/projected/2a88405b-f7ae-4ed2-b039-65f61dff1c01-kube-api-access-k8gs4\") pod \"kuadrant-operator-controller-manager-6bd956ccf-fpbkf\" (UID: \"2a88405b-f7ae-4ed2-b039-65f61dff1c01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-fpbkf" Jul 31 18:51:01.628992 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.628875 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a88405b-f7ae-4ed2-b039-65f61dff1c01-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-fpbkf\" (UID: \"2a88405b-f7ae-4ed2-b039-65f61dff1c01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-fpbkf" Jul 31 18:51:01.637547 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:51:01.637515 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-fczlj\" is forbidden: User \"system:node:ip-10-0-139-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-39.ec2.internal' and this object" podUID="9d3a24b1-2a71-4af7-92f6-6af1eb2c2d01" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fczlj" Jul 31 18:51:01.729743 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.729706 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8gs4\" (UniqueName: \"kubernetes.io/projected/2a88405b-f7ae-4ed2-b039-65f61dff1c01-kube-api-access-k8gs4\") pod \"kuadrant-operator-controller-manager-6bd956ccf-fpbkf\" (UID: \"2a88405b-f7ae-4ed2-b039-65f61dff1c01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-fpbkf" Jul 31 18:51:01.729887 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.729778 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a88405b-f7ae-4ed2-b039-65f61dff1c01-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-fpbkf\" (UID: \"2a88405b-f7ae-4ed2-b039-65f61dff1c01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-fpbkf" Jul 31 18:51:01.730161 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.730135 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a88405b-f7ae-4ed2-b039-65f61dff1c01-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-fpbkf\" (UID: \"2a88405b-f7ae-4ed2-b039-65f61dff1c01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-fpbkf" Jul 31 18:51:01.746073 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:51:01.746043 2565 log.go:32] "RunPodSandbox from runtime service failed" err=< Jul 31 18:51:01.746073 ip-10-0-139-39 kubenswrapper[2565]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6bd956ccf-4djvz_kuadrant-system_70aa0976-4b52-4070-8790-d799f5871e2c_0(8bf6c32bc2b7c36ebef383706d5e40f35cc217152b955a5647f9d488acf4a48b): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6bd956ccf-4djvz to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"8bf6c32bc2b7c36ebef383706d5e40f35cc217152b955a5647f9d488acf4a48b" Netns:"/var/run/netns/49889e73-2ca8-4b9e-96ae-3bfaec72d180" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-6bd956ccf-4djvz;K8S_POD_INFRA_CONTAINER_ID=8bf6c32bc2b7c36ebef383706d5e40f35cc217152b955a5647f9d488acf4a48b;K8S_POD_UID=70aa0976-4b52-4070-8790-d799f5871e2c" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-4djvz] networking: [kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-4djvz/70aa0976-4b52-4070-8790-d799f5871e2c:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-4djvz 8bf6c32bc2b7c36ebef383706d5e40f35cc217152b955a5647f9d488acf4a48b network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:85:00:25 [10.133.0.37/23] Jul 31 18:51:01.746073 ip-10-0-139-39 kubenswrapper[2565]: ' Jul 31 18:51:01.746073 ip-10-0-139-39 kubenswrapper[2565]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 31 18:51:01.746073 ip-10-0-139-39 kubenswrapper[2565]: > Jul 31 18:51:01.746332 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.746085 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8gs4\" (UniqueName: \"kubernetes.io/projected/2a88405b-f7ae-4ed2-b039-65f61dff1c01-kube-api-access-k8gs4\") pod \"kuadrant-operator-controller-manager-6bd956ccf-fpbkf\" (UID: \"2a88405b-f7ae-4ed2-b039-65f61dff1c01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-fpbkf" Jul 31 18:51:01.746332 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:51:01.746094 2565 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Jul 31 18:51:01.746332 ip-10-0-139-39 kubenswrapper[2565]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6bd956ccf-4djvz_kuadrant-system_70aa0976-4b52-4070-8790-d799f5871e2c_0(8bf6c32bc2b7c36ebef383706d5e40f35cc217152b955a5647f9d488acf4a48b): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6bd956ccf-4djvz to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"8bf6c32bc2b7c36ebef383706d5e40f35cc217152b955a5647f9d488acf4a48b" Netns:"/var/run/netns/49889e73-2ca8-4b9e-96ae-3bfaec72d180" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-6bd956ccf-4djvz;K8S_POD_INFRA_CONTAINER_ID=8bf6c32bc2b7c36ebef383706d5e40f35cc217152b955a5647f9d488acf4a48b;K8S_POD_UID=70aa0976-4b52-4070-8790-d799f5871e2c" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-4djvz] networking: [kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-4djvz/70aa0976-4b52-4070-8790-d799f5871e2c:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-4djvz 8bf6c32bc2b7c36ebef383706d5e40f35cc217152b955a5647f9d488acf4a48b network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:85:00:25 [10.133.0.37/23] Jul 31 18:51:01.746332 ip-10-0-139-39 kubenswrapper[2565]: ' Jul 31 18:51:01.746332 ip-10-0-139-39 kubenswrapper[2565]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 31 18:51:01.746332 ip-10-0-139-39 kubenswrapper[2565]: > pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-4djvz" Jul 31 18:51:01.761858 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.761831 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fczlj" Jul 31 18:51:01.764434 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:51:01.764407 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-fczlj\" is forbidden: User \"system:node:ip-10-0-139-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-39.ec2.internal' and this object" podUID="9d3a24b1-2a71-4af7-92f6-6af1eb2c2d01" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fczlj" Jul 31 18:51:01.818880 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.818851 2565 generic.go:356] "Generic (PLEG): container finished" podID="9d3a24b1-2a71-4af7-92f6-6af1eb2c2d01" containerID="e2a63784d5a5c4fd92b7a3fccb98a1e2c77908dc0658e82d410919fd9c2ad73d" exitCode=0 Jul 31 18:51:01.819062 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.818895 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fczlj" Jul 31 18:51:01.819062 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.818904 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-4djvz" Jul 31 18:51:01.819062 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.818949 2565 scope.go:117] "RemoveContainer" containerID="e2a63784d5a5c4fd92b7a3fccb98a1e2c77908dc0658e82d410919fd9c2ad73d" Jul 31 18:51:01.820720 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:51:01.820695 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6bd956ccf-4djvz\" is forbidden: User \"system:node:ip-10-0-139-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-39.ec2.internal' and this object" podUID="70aa0976-4b52-4070-8790-d799f5871e2c" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-4djvz" Jul 31 18:51:01.822298 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:51:01.822276 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-fczlj\" is forbidden: User \"system:node:ip-10-0-139-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-39.ec2.internal' and this object" podUID="9d3a24b1-2a71-4af7-92f6-6af1eb2c2d01" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fczlj" Jul 31 18:51:01.823756 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:51:01.823730 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6bd956ccf-4djvz\" is forbidden: User \"system:node:ip-10-0-139-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-39.ec2.internal' and this object" podUID="70aa0976-4b52-4070-8790-d799f5871e2c" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-4djvz" Jul 31 18:51:01.825339 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:51:01.825310 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-fczlj\" is forbidden: User \"system:node:ip-10-0-139-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-39.ec2.internal' and this object" podUID="9d3a24b1-2a71-4af7-92f6-6af1eb2c2d01" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fczlj" Jul 31 18:51:01.827172 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.827151 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-4djvz" Jul 31 18:51:01.827770 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.827755 2565 scope.go:117] "RemoveContainer" containerID="e2a63784d5a5c4fd92b7a3fccb98a1e2c77908dc0658e82d410919fd9c2ad73d" Jul 31 18:51:01.828026 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:51:01.828007 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2a63784d5a5c4fd92b7a3fccb98a1e2c77908dc0658e82d410919fd9c2ad73d\": container with ID starting with e2a63784d5a5c4fd92b7a3fccb98a1e2c77908dc0658e82d410919fd9c2ad73d not found: ID does not exist" containerID="e2a63784d5a5c4fd92b7a3fccb98a1e2c77908dc0658e82d410919fd9c2ad73d" Jul 31 18:51:01.828098 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.828033 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2a63784d5a5c4fd92b7a3fccb98a1e2c77908dc0658e82d410919fd9c2ad73d"} err="failed to get container status \"e2a63784d5a5c4fd92b7a3fccb98a1e2c77908dc0658e82d410919fd9c2ad73d\": rpc error: code = NotFound desc = could not find container \"e2a63784d5a5c4fd92b7a3fccb98a1e2c77908dc0658e82d410919fd9c2ad73d\": container with ID starting with e2a63784d5a5c4fd92b7a3fccb98a1e2c77908dc0658e82d410919fd9c2ad73d not found: ID does not exist" Jul 31 18:51:01.828985 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:51:01.828963 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6bd956ccf-4djvz\" is forbidden: User \"system:node:ip-10-0-139-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-39.ec2.internal' and this object" podUID="70aa0976-4b52-4070-8790-d799f5871e2c" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-4djvz" Jul 31 18:51:01.830458 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:51:01.830436 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-fczlj\" is forbidden: User \"system:node:ip-10-0-139-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-39.ec2.internal' and this object" podUID="9d3a24b1-2a71-4af7-92f6-6af1eb2c2d01" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fczlj" Jul 31 18:51:01.891853 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.891829 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-fpbkf" Jul 31 18:51:01.931445 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.931366 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmnn4\" (UniqueName: \"kubernetes.io/projected/70aa0976-4b52-4070-8790-d799f5871e2c-kube-api-access-kmnn4\") pod \"70aa0976-4b52-4070-8790-d799f5871e2c\" (UID: \"70aa0976-4b52-4070-8790-d799f5871e2c\") " Jul 31 18:51:01.931445 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.931433 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tg8pz\" (UniqueName: \"kubernetes.io/projected/9d3a24b1-2a71-4af7-92f6-6af1eb2c2d01-kube-api-access-tg8pz\") pod \"9d3a24b1-2a71-4af7-92f6-6af1eb2c2d01\" (UID: \"9d3a24b1-2a71-4af7-92f6-6af1eb2c2d01\") " Jul 31 18:51:01.931677 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.931492 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70aa0976-4b52-4070-8790-d799f5871e2c-extensions-socket-volume\") pod \"70aa0976-4b52-4070-8790-d799f5871e2c\" (UID: \"70aa0976-4b52-4070-8790-d799f5871e2c\") " Jul 31 18:51:01.931794 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.931767 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70aa0976-4b52-4070-8790-d799f5871e2c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "70aa0976-4b52-4070-8790-d799f5871e2c" (UID: "70aa0976-4b52-4070-8790-d799f5871e2c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 18:51:01.933461 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.933437 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70aa0976-4b52-4070-8790-d799f5871e2c-kube-api-access-kmnn4" (OuterVolumeSpecName: "kube-api-access-kmnn4") pod "70aa0976-4b52-4070-8790-d799f5871e2c" (UID: "70aa0976-4b52-4070-8790-d799f5871e2c"). InnerVolumeSpecName "kube-api-access-kmnn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 18:51:01.933571 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:01.933552 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d3a24b1-2a71-4af7-92f6-6af1eb2c2d01-kube-api-access-tg8pz" (OuterVolumeSpecName: "kube-api-access-tg8pz") pod "9d3a24b1-2a71-4af7-92f6-6af1eb2c2d01" (UID: "9d3a24b1-2a71-4af7-92f6-6af1eb2c2d01"). InnerVolumeSpecName "kube-api-access-tg8pz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 18:51:02.032456 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:02.032421 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmnn4\" (UniqueName: \"kubernetes.io/projected/70aa0976-4b52-4070-8790-d799f5871e2c-kube-api-access-kmnn4\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:51:02.032456 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:02.032449 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg8pz\" (UniqueName: \"kubernetes.io/projected/9d3a24b1-2a71-4af7-92f6-6af1eb2c2d01-kube-api-access-tg8pz\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:51:02.032456 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:02.032461 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70aa0976-4b52-4070-8790-d799f5871e2c-extensions-socket-volume\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:51:02.036231 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:02.036206 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-fpbkf"] Jul 31 18:51:02.038667 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:51:02.038639 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a88405b_f7ae_4ed2_b039_65f61dff1c01.slice/crio-81f2c75e96bed6e666a2b1396bcd59b3bfd5f55bd0e6e3a2a07542bcba9370b6 WatchSource:0}: Error finding container 81f2c75e96bed6e666a2b1396bcd59b3bfd5f55bd0e6e3a2a07542bcba9370b6: Status 404 returned error can't find the container with id 81f2c75e96bed6e666a2b1396bcd59b3bfd5f55bd0e6e3a2a07542bcba9370b6 Jul 31 18:51:02.826519 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:02.826462 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-fpbkf" event={"ID":"2a88405b-f7ae-4ed2-b039-65f61dff1c01","Type":"ContainerStarted","Data":"81f2c75e96bed6e666a2b1396bcd59b3bfd5f55bd0e6e3a2a07542bcba9370b6"} Jul 31 18:51:02.827709 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:02.827679 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-4djvz" Jul 31 18:51:02.837845 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:51:02.837816 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6bd956ccf-4djvz\" is forbidden: User \"system:node:ip-10-0-139-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-39.ec2.internal' and this object" podUID="70aa0976-4b52-4070-8790-d799f5871e2c" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-4djvz" Jul 31 18:51:03.884897 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:51:03.884857 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6bd956ccf-4djvz\" is forbidden: User \"system:node:ip-10-0-139-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-39.ec2.internal' and this object" podUID="70aa0976-4b52-4070-8790-d799f5871e2c" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-4djvz" Jul 31 18:51:03.887596 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:03.887569 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70aa0976-4b52-4070-8790-d799f5871e2c" path="/var/lib/kubelet/pods/70aa0976-4b52-4070-8790-d799f5871e2c/volumes" Jul 31 18:51:03.887888 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:03.887871 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d3a24b1-2a71-4af7-92f6-6af1eb2c2d01" path="/var/lib/kubelet/pods/9d3a24b1-2a71-4af7-92f6-6af1eb2c2d01/volumes" Jul 31 18:51:04.093334 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:04.093281 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:51:04.196967 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:04.196889 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:51:05.844284 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:05.844181 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-fpbkf" event={"ID":"2a88405b-f7ae-4ed2-b039-65f61dff1c01","Type":"ContainerStarted","Data":"62fe6c44efe480b9fe12bcc2a819485ed763feea17512ba99fac91a79bacf102"} Jul 31 18:51:05.844688 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:05.844355 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-fpbkf" Jul 31 18:51:05.867387 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:05.867331 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-fpbkf" podStartSLOduration=1.317759941 podStartE2EDuration="4.867313617s" totalImagesPullingTime="3.549553676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:51:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:51:05.866466949 +0000 UTC m=+442.504456748" watchObservedRunningTime="2026-07-31 18:51:05.867313617 +0000 UTC m=+442.505303421" Jul 31 18:51:06.712300 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:06.712215 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:51:06.811951 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:06.811917 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:51:10.852955 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:10.852903 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5648497788-ctvkm" podUID="4a296f33-4197-44b6-8a5e-a1ae0c83dfe8" containerName="console" containerID="cri-o://b977e9da66d9ecc8e18dcdd13d25a532ba28fced6098c5bacd1edc9f6ba083b7" gracePeriod=15 Jul 31 18:51:11.156446 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.156421 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5648497788-ctvkm" Jul 31 18:51:11.317475 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.317448 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcvtp\" (UniqueName: \"kubernetes.io/projected/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-kube-api-access-lcvtp\") pod \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\" (UID: \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\") " Jul 31 18:51:11.317642 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.317502 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-oauth-serving-cert\") pod \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\" (UID: \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\") " Jul 31 18:51:11.317642 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.317522 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-console-config\") pod \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\" (UID: \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\") " Jul 31 18:51:11.317642 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.317542 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-trusted-ca-bundle\") pod \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\" (UID: \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\") " Jul 31 18:51:11.317642 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.317577 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-service-ca\") pod \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\" (UID: \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\") " Jul 31 18:51:11.317863 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.317836 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-console-oauth-config\") pod \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\" (UID: \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\") " Jul 31 18:51:11.318001 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.317891 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-console-serving-cert\") pod \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\" (UID: \"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8\") " Jul 31 18:51:11.318001 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.317929 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4a296f33-4197-44b6-8a5e-a1ae0c83dfe8" (UID: "4a296f33-4197-44b6-8a5e-a1ae0c83dfe8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 18:51:11.318125 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.318010 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-service-ca" (OuterVolumeSpecName: "service-ca") pod "4a296f33-4197-44b6-8a5e-a1ae0c83dfe8" (UID: "4a296f33-4197-44b6-8a5e-a1ae0c83dfe8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 18:51:11.318125 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.318016 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-console-config" (OuterVolumeSpecName: "console-config") pod "4a296f33-4197-44b6-8a5e-a1ae0c83dfe8" (UID: "4a296f33-4197-44b6-8a5e-a1ae0c83dfe8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 18:51:11.318125 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.318059 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4a296f33-4197-44b6-8a5e-a1ae0c83dfe8" (UID: "4a296f33-4197-44b6-8a5e-a1ae0c83dfe8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 18:51:11.318295 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.318229 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-service-ca\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:51:11.318295 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.318279 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-oauth-serving-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:51:11.318377 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.318294 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-console-config\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:51:11.318377 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.318308 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-trusted-ca-bundle\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:51:11.319733 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.319706 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-kube-api-access-lcvtp" (OuterVolumeSpecName: "kube-api-access-lcvtp") pod "4a296f33-4197-44b6-8a5e-a1ae0c83dfe8" (UID: "4a296f33-4197-44b6-8a5e-a1ae0c83dfe8"). InnerVolumeSpecName "kube-api-access-lcvtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 18:51:11.319926 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.319899 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4a296f33-4197-44b6-8a5e-a1ae0c83dfe8" (UID: "4a296f33-4197-44b6-8a5e-a1ae0c83dfe8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:51:11.319974 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.319913 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4a296f33-4197-44b6-8a5e-a1ae0c83dfe8" (UID: "4a296f33-4197-44b6-8a5e-a1ae0c83dfe8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:51:11.419311 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.419221 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-console-oauth-config\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:51:11.419311 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.419270 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-console-serving-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:51:11.419311 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.419285 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcvtp\" (UniqueName: \"kubernetes.io/projected/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8-kube-api-access-lcvtp\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:51:11.620441 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.620405 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:51:11.723639 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.723559 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:51:11.820998 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.820966 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:51:11.877446 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.877414 2565 generic.go:356] "Generic (PLEG): container finished" podID="4a296f33-4197-44b6-8a5e-a1ae0c83dfe8" containerID="b977e9da66d9ecc8e18dcdd13d25a532ba28fced6098c5bacd1edc9f6ba083b7" exitCode=0 Jul 31 18:51:11.877893 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.877466 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5648497788-ctvkm" event={"ID":"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8","Type":"ContainerDied","Data":"b977e9da66d9ecc8e18dcdd13d25a532ba28fced6098c5bacd1edc9f6ba083b7"} Jul 31 18:51:11.877893 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.877477 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5648497788-ctvkm" Jul 31 18:51:11.877893 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.877527 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5648497788-ctvkm" event={"ID":"4a296f33-4197-44b6-8a5e-a1ae0c83dfe8","Type":"ContainerDied","Data":"2d804bce73a727f93297965a0e46346a8b68e490d4e38589059371acfd9ff823"} Jul 31 18:51:11.877893 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.877547 2565 scope.go:117] "RemoveContainer" containerID="b977e9da66d9ecc8e18dcdd13d25a532ba28fced6098c5bacd1edc9f6ba083b7" Jul 31 18:51:11.887020 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.886870 2565 scope.go:117] "RemoveContainer" containerID="b977e9da66d9ecc8e18dcdd13d25a532ba28fced6098c5bacd1edc9f6ba083b7" Jul 31 18:51:11.887171 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:51:11.887145 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b977e9da66d9ecc8e18dcdd13d25a532ba28fced6098c5bacd1edc9f6ba083b7\": container with ID starting with b977e9da66d9ecc8e18dcdd13d25a532ba28fced6098c5bacd1edc9f6ba083b7 not found: ID does not exist" containerID="b977e9da66d9ecc8e18dcdd13d25a532ba28fced6098c5bacd1edc9f6ba083b7" Jul 31 18:51:11.887299 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.887179 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b977e9da66d9ecc8e18dcdd13d25a532ba28fced6098c5bacd1edc9f6ba083b7"} err="failed to get container status \"b977e9da66d9ecc8e18dcdd13d25a532ba28fced6098c5bacd1edc9f6ba083b7\": rpc error: code = NotFound desc = could not find container \"b977e9da66d9ecc8e18dcdd13d25a532ba28fced6098c5bacd1edc9f6ba083b7\": container with ID starting with b977e9da66d9ecc8e18dcdd13d25a532ba28fced6098c5bacd1edc9f6ba083b7 not found: ID does not exist" Jul 31 18:51:11.902633 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.902609 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5648497788-ctvkm"] Jul 31 18:51:11.908733 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:11.908708 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5648497788-ctvkm"] Jul 31 18:51:13.886912 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:13.886875 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a296f33-4197-44b6-8a5e-a1ae0c83dfe8" path="/var/lib/kubelet/pods/4a296f33-4197-44b6-8a5e-a1ae0c83dfe8/volumes" Jul 31 18:51:16.825518 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:16.825483 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:51:16.851063 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:16.851035 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-fpbkf" Jul 31 18:51:16.978286 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:16.978208 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:51:17.132822 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:17.132732 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:51:17.380545 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:17.380507 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bftdk"] Jul 31 18:51:17.380880 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:17.380868 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a296f33-4197-44b6-8a5e-a1ae0c83dfe8" containerName="console" Jul 31 18:51:17.380923 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:17.380883 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a296f33-4197-44b6-8a5e-a1ae0c83dfe8" containerName="console" Jul 31 18:51:17.380923 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:17.380898 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d3a24b1-2a71-4af7-92f6-6af1eb2c2d01" containerName="manager" Jul 31 18:51:17.380923 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:17.380905 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d3a24b1-2a71-4af7-92f6-6af1eb2c2d01" containerName="manager" Jul 31 18:51:17.381012 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:17.380967 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a296f33-4197-44b6-8a5e-a1ae0c83dfe8" containerName="console" Jul 31 18:51:17.381012 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:17.380977 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d3a24b1-2a71-4af7-92f6-6af1eb2c2d01" containerName="manager" Jul 31 18:51:17.387647 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:17.387600 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bftdk" Jul 31 18:51:17.398948 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:17.398923 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bftdk"] Jul 31 18:51:17.466354 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:17.466325 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ftlk\" (UniqueName: \"kubernetes.io/projected/ec504d44-ecbb-41dc-9aa6-34e58c000207-kube-api-access-7ftlk\") pod \"kuadrant-operator-controller-manager-68cf589589-bftdk\" (UID: \"ec504d44-ecbb-41dc-9aa6-34e58c000207\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bftdk" Jul 31 18:51:17.466517 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:17.466382 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec504d44-ecbb-41dc-9aa6-34e58c000207-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-bftdk\" (UID: \"ec504d44-ecbb-41dc-9aa6-34e58c000207\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bftdk" Jul 31 18:51:17.567183 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:17.567141 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec504d44-ecbb-41dc-9aa6-34e58c000207-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-bftdk\" (UID: \"ec504d44-ecbb-41dc-9aa6-34e58c000207\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bftdk" Jul 31 18:51:17.567389 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:17.567264 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ftlk\" (UniqueName: \"kubernetes.io/projected/ec504d44-ecbb-41dc-9aa6-34e58c000207-kube-api-access-7ftlk\") pod \"kuadrant-operator-controller-manager-68cf589589-bftdk\" (UID: \"ec504d44-ecbb-41dc-9aa6-34e58c000207\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bftdk" Jul 31 18:51:17.567580 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:17.567558 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec504d44-ecbb-41dc-9aa6-34e58c000207-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-bftdk\" (UID: \"ec504d44-ecbb-41dc-9aa6-34e58c000207\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bftdk" Jul 31 18:51:17.578309 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:17.578281 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ftlk\" (UniqueName: \"kubernetes.io/projected/ec504d44-ecbb-41dc-9aa6-34e58c000207-kube-api-access-7ftlk\") pod \"kuadrant-operator-controller-manager-68cf589589-bftdk\" (UID: \"ec504d44-ecbb-41dc-9aa6-34e58c000207\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bftdk" Jul 31 18:51:17.697268 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:17.697150 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bftdk" Jul 31 18:51:17.839394 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:17.839368 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bftdk"] Jul 31 18:51:17.842198 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:51:17.842172 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec504d44_ecbb_41dc_9aa6_34e58c000207.slice/crio-a3953ba4ab47b0ce888d07d914f6f8cec81d4b45961d4f53fadc8a0993f42d96 WatchSource:0}: Error finding container a3953ba4ab47b0ce888d07d914f6f8cec81d4b45961d4f53fadc8a0993f42d96: Status 404 returned error can't find the container with id a3953ba4ab47b0ce888d07d914f6f8cec81d4b45961d4f53fadc8a0993f42d96 Jul 31 18:51:17.907788 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:17.907763 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bftdk" event={"ID":"ec504d44-ecbb-41dc-9aa6-34e58c000207","Type":"ContainerStarted","Data":"a3953ba4ab47b0ce888d07d914f6f8cec81d4b45961d4f53fadc8a0993f42d96"} Jul 31 18:51:18.914210 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:18.914175 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bftdk" event={"ID":"ec504d44-ecbb-41dc-9aa6-34e58c000207","Type":"ContainerStarted","Data":"e639fea3b10027ee73576d4489e61030a6c58a6732276b6497ca28dd82b6cdeb"} Jul 31 18:51:18.914622 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:18.914262 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bftdk" Jul 31 18:51:18.949721 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:18.949653 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bftdk" podStartSLOduration=1.949635802 podStartE2EDuration="1.949635802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:51:18.949446022 +0000 UTC m=+455.587435823" watchObservedRunningTime="2026-07-31 18:51:18.949635802 +0000 UTC m=+455.587625603" Jul 31 18:51:29.921445 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:29.921410 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bftdk" Jul 31 18:51:29.991795 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:29.991760 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-fpbkf"] Jul 31 18:51:29.991967 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:29.991947 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-fpbkf" podUID="2a88405b-f7ae-4ed2-b039-65f61dff1c01" containerName="manager" containerID="cri-o://62fe6c44efe480b9fe12bcc2a819485ed763feea17512ba99fac91a79bacf102" gracePeriod=10 Jul 31 18:51:30.239154 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:30.239126 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-fpbkf" Jul 31 18:51:30.269861 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:30.269825 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8gs4\" (UniqueName: \"kubernetes.io/projected/2a88405b-f7ae-4ed2-b039-65f61dff1c01-kube-api-access-k8gs4\") pod \"2a88405b-f7ae-4ed2-b039-65f61dff1c01\" (UID: \"2a88405b-f7ae-4ed2-b039-65f61dff1c01\") " Jul 31 18:51:30.270056 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:30.269946 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a88405b-f7ae-4ed2-b039-65f61dff1c01-extensions-socket-volume\") pod \"2a88405b-f7ae-4ed2-b039-65f61dff1c01\" (UID: \"2a88405b-f7ae-4ed2-b039-65f61dff1c01\") " Jul 31 18:51:30.270484 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:30.270452 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a88405b-f7ae-4ed2-b039-65f61dff1c01-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2a88405b-f7ae-4ed2-b039-65f61dff1c01" (UID: "2a88405b-f7ae-4ed2-b039-65f61dff1c01"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 18:51:30.272012 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:30.271989 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a88405b-f7ae-4ed2-b039-65f61dff1c01-kube-api-access-k8gs4" (OuterVolumeSpecName: "kube-api-access-k8gs4") pod "2a88405b-f7ae-4ed2-b039-65f61dff1c01" (UID: "2a88405b-f7ae-4ed2-b039-65f61dff1c01"). InnerVolumeSpecName "kube-api-access-k8gs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 18:51:30.371533 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:30.371497 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a88405b-f7ae-4ed2-b039-65f61dff1c01-extensions-socket-volume\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:51:30.371533 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:30.371529 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8gs4\" (UniqueName: \"kubernetes.io/projected/2a88405b-f7ae-4ed2-b039-65f61dff1c01-kube-api-access-k8gs4\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:51:30.977821 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:30.977783 2565 generic.go:356] "Generic (PLEG): container finished" podID="2a88405b-f7ae-4ed2-b039-65f61dff1c01" containerID="62fe6c44efe480b9fe12bcc2a819485ed763feea17512ba99fac91a79bacf102" exitCode=0 Jul 31 18:51:30.978283 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:30.977831 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-fpbkf" Jul 31 18:51:30.978283 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:30.977851 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-fpbkf" event={"ID":"2a88405b-f7ae-4ed2-b039-65f61dff1c01","Type":"ContainerDied","Data":"62fe6c44efe480b9fe12bcc2a819485ed763feea17512ba99fac91a79bacf102"} Jul 31 18:51:30.978283 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:30.977895 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-fpbkf" event={"ID":"2a88405b-f7ae-4ed2-b039-65f61dff1c01","Type":"ContainerDied","Data":"81f2c75e96bed6e666a2b1396bcd59b3bfd5f55bd0e6e3a2a07542bcba9370b6"} Jul 31 18:51:30.978283 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:30.977919 2565 scope.go:117] "RemoveContainer" containerID="62fe6c44efe480b9fe12bcc2a819485ed763feea17512ba99fac91a79bacf102" Jul 31 18:51:30.986700 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:30.986677 2565 scope.go:117] "RemoveContainer" containerID="62fe6c44efe480b9fe12bcc2a819485ed763feea17512ba99fac91a79bacf102" Jul 31 18:51:30.986965 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:51:30.986948 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62fe6c44efe480b9fe12bcc2a819485ed763feea17512ba99fac91a79bacf102\": container with ID starting with 62fe6c44efe480b9fe12bcc2a819485ed763feea17512ba99fac91a79bacf102 not found: ID does not exist" containerID="62fe6c44efe480b9fe12bcc2a819485ed763feea17512ba99fac91a79bacf102" Jul 31 18:51:30.987009 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:30.986973 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62fe6c44efe480b9fe12bcc2a819485ed763feea17512ba99fac91a79bacf102"} err="failed to get container status \"62fe6c44efe480b9fe12bcc2a819485ed763feea17512ba99fac91a79bacf102\": rpc error: code = NotFound desc = could not find container \"62fe6c44efe480b9fe12bcc2a819485ed763feea17512ba99fac91a79bacf102\": container with ID starting with 62fe6c44efe480b9fe12bcc2a819485ed763feea17512ba99fac91a79bacf102 not found: ID does not exist" Jul 31 18:51:31.005184 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:31.005156 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-fpbkf"] Jul 31 18:51:31.009029 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:31.008998 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-fpbkf"] Jul 31 18:51:31.885962 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:31.885916 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a88405b-f7ae-4ed2-b039-65f61dff1c01" path="/var/lib/kubelet/pods/2a88405b-f7ae-4ed2-b039-65f61dff1c01/volumes" Jul 31 18:51:33.366876 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.366847 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k"] Jul 31 18:51:33.367211 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.367201 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a88405b-f7ae-4ed2-b039-65f61dff1c01" containerName="manager" Jul 31 18:51:33.367284 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.367213 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a88405b-f7ae-4ed2-b039-65f61dff1c01" containerName="manager" Jul 31 18:51:33.367321 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.367309 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a88405b-f7ae-4ed2-b039-65f61dff1c01" containerName="manager" Jul 31 18:51:33.370009 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.369990 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:33.372159 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.372140 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-hv2rx\"" Jul 31 18:51:33.382057 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.382025 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k"] Jul 31 18:51:33.395650 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.395587 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:51:33.493413 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.493364 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:51:33.498537 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.498518 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed4e23e4-7b4d-4214-a571-0f54d841c7c5-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w8k4k\" (UID: \"ed4e23e4-7b4d-4214-a571-0f54d841c7c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:33.498612 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.498557 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed4e23e4-7b4d-4214-a571-0f54d841c7c5-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w8k4k\" (UID: \"ed4e23e4-7b4d-4214-a571-0f54d841c7c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:33.498612 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.498581 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mg4q\" (UniqueName: \"kubernetes.io/projected/ed4e23e4-7b4d-4214-a571-0f54d841c7c5-kube-api-access-4mg4q\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w8k4k\" (UID: \"ed4e23e4-7b4d-4214-a571-0f54d841c7c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:33.498718 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.498648 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed4e23e4-7b4d-4214-a571-0f54d841c7c5-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w8k4k\" (UID: \"ed4e23e4-7b4d-4214-a571-0f54d841c7c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:33.498718 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.498675 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed4e23e4-7b4d-4214-a571-0f54d841c7c5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w8k4k\" (UID: \"ed4e23e4-7b4d-4214-a571-0f54d841c7c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:33.498718 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.498699 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed4e23e4-7b4d-4214-a571-0f54d841c7c5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w8k4k\" (UID: \"ed4e23e4-7b4d-4214-a571-0f54d841c7c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:33.498842 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.498726 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed4e23e4-7b4d-4214-a571-0f54d841c7c5-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w8k4k\" (UID: \"ed4e23e4-7b4d-4214-a571-0f54d841c7c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:33.498842 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.498752 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed4e23e4-7b4d-4214-a571-0f54d841c7c5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w8k4k\" (UID: \"ed4e23e4-7b4d-4214-a571-0f54d841c7c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:33.498933 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.498835 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed4e23e4-7b4d-4214-a571-0f54d841c7c5-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w8k4k\" (UID: \"ed4e23e4-7b4d-4214-a571-0f54d841c7c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:33.599570 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.599539 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed4e23e4-7b4d-4214-a571-0f54d841c7c5-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w8k4k\" (UID: \"ed4e23e4-7b4d-4214-a571-0f54d841c7c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:33.599706 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.599587 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed4e23e4-7b4d-4214-a571-0f54d841c7c5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w8k4k\" (UID: \"ed4e23e4-7b4d-4214-a571-0f54d841c7c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:33.599706 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.599620 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed4e23e4-7b4d-4214-a571-0f54d841c7c5-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w8k4k\" (UID: \"ed4e23e4-7b4d-4214-a571-0f54d841c7c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:33.599706 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.599676 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed4e23e4-7b4d-4214-a571-0f54d841c7c5-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w8k4k\" (UID: \"ed4e23e4-7b4d-4214-a571-0f54d841c7c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:33.599832 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.599731 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed4e23e4-7b4d-4214-a571-0f54d841c7c5-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w8k4k\" (UID: \"ed4e23e4-7b4d-4214-a571-0f54d841c7c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:33.599832 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.599765 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mg4q\" (UniqueName: \"kubernetes.io/projected/ed4e23e4-7b4d-4214-a571-0f54d841c7c5-kube-api-access-4mg4q\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w8k4k\" (UID: \"ed4e23e4-7b4d-4214-a571-0f54d841c7c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:33.599832 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.599820 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed4e23e4-7b4d-4214-a571-0f54d841c7c5-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w8k4k\" (UID: \"ed4e23e4-7b4d-4214-a571-0f54d841c7c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:33.599975 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.599858 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed4e23e4-7b4d-4214-a571-0f54d841c7c5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w8k4k\" (UID: \"ed4e23e4-7b4d-4214-a571-0f54d841c7c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:33.599975 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.599891 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed4e23e4-7b4d-4214-a571-0f54d841c7c5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w8k4k\" (UID: \"ed4e23e4-7b4d-4214-a571-0f54d841c7c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:33.600133 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.600101 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed4e23e4-7b4d-4214-a571-0f54d841c7c5-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w8k4k\" (UID: \"ed4e23e4-7b4d-4214-a571-0f54d841c7c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:33.600201 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.600177 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed4e23e4-7b4d-4214-a571-0f54d841c7c5-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w8k4k\" (UID: \"ed4e23e4-7b4d-4214-a571-0f54d841c7c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:33.600302 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.600279 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed4e23e4-7b4d-4214-a571-0f54d841c7c5-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w8k4k\" (UID: \"ed4e23e4-7b4d-4214-a571-0f54d841c7c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:33.600448 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.600420 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed4e23e4-7b4d-4214-a571-0f54d841c7c5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w8k4k\" (UID: \"ed4e23e4-7b4d-4214-a571-0f54d841c7c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:33.600578 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.600423 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed4e23e4-7b4d-4214-a571-0f54d841c7c5-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w8k4k\" (UID: \"ed4e23e4-7b4d-4214-a571-0f54d841c7c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:33.602584 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.602564 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed4e23e4-7b4d-4214-a571-0f54d841c7c5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w8k4k\" (UID: \"ed4e23e4-7b4d-4214-a571-0f54d841c7c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:33.602701 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.602666 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed4e23e4-7b4d-4214-a571-0f54d841c7c5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w8k4k\" (UID: \"ed4e23e4-7b4d-4214-a571-0f54d841c7c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:33.607701 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.607679 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed4e23e4-7b4d-4214-a571-0f54d841c7c5-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w8k4k\" (UID: \"ed4e23e4-7b4d-4214-a571-0f54d841c7c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:33.607936 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.607918 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mg4q\" (UniqueName: \"kubernetes.io/projected/ed4e23e4-7b4d-4214-a571-0f54d841c7c5-kube-api-access-4mg4q\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w8k4k\" (UID: \"ed4e23e4-7b4d-4214-a571-0f54d841c7c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:33.681228 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.681161 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:33.815292 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.815268 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k"] Jul 31 18:51:33.817020 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:51:33.816994 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded4e23e4_7b4d_4214_a571_0f54d841c7c5.slice/crio-921a785a59f72d7e97b0a7791c746cf938b5543f19fcc971261cf25a3159102b WatchSource:0}: Error finding container 921a785a59f72d7e97b0a7791c746cf938b5543f19fcc971261cf25a3159102b: Status 404 returned error can't find the container with id 921a785a59f72d7e97b0a7791c746cf938b5543f19fcc971261cf25a3159102b Jul 31 18:51:33.819663 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.819633 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 18:51:33.819734 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.819700 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 18:51:33.819734 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.819730 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 18:51:33.993593 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.993501 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" event={"ID":"ed4e23e4-7b4d-4214-a571-0f54d841c7c5","Type":"ContainerStarted","Data":"76263e46c83ba6db6bf7c26521b6a601eff9d3d6ddd4efce1b15c153e04f5bbd"} Jul 31 18:51:33.993593 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:33.993569 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" event={"ID":"ed4e23e4-7b4d-4214-a571-0f54d841c7c5","Type":"ContainerStarted","Data":"921a785a59f72d7e97b0a7791c746cf938b5543f19fcc971261cf25a3159102b"} Jul 31 18:51:34.014986 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:34.014936 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" podStartSLOduration=1.014920431 podStartE2EDuration="1.014920431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:51:34.013999163 +0000 UTC m=+470.651988962" watchObservedRunningTime="2026-07-31 18:51:34.014920431 +0000 UTC m=+470.652910232" Jul 31 18:51:34.681726 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:34.681692 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:34.686413 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:34.686391 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:34.998331 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:34.998301 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:34.999198 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:34.999178 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:51:51.344911 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:51.344875 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:51:51.453652 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:51.453605 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:51:51.517135 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:51.517100 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:51:51.768649 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:51.768619 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 18:51:51.771951 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:51.771929 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb" Jul 31 18:51:51.774100 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:51.774080 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 31 18:51:51.781027 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:51.781001 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 18:51:51.795315 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:51.795286 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 18:51:51.867123 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:51.867098 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/90363ad8-673b-45b6-887f-c82f70632eba-config-file\") pod \"limitador-limitador-85d9db4d5d-h7vrb\" (UID: \"90363ad8-673b-45b6-887f-c82f70632eba\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb" Jul 31 18:51:51.867279 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:51.867132 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dbc9\" (UniqueName: \"kubernetes.io/projected/90363ad8-673b-45b6-887f-c82f70632eba-kube-api-access-2dbc9\") pod \"limitador-limitador-85d9db4d5d-h7vrb\" (UID: \"90363ad8-673b-45b6-887f-c82f70632eba\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb" Jul 31 18:51:51.967929 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:51.967897 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/90363ad8-673b-45b6-887f-c82f70632eba-config-file\") pod \"limitador-limitador-85d9db4d5d-h7vrb\" (UID: \"90363ad8-673b-45b6-887f-c82f70632eba\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb" Jul 31 18:51:51.967929 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:51.967936 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dbc9\" (UniqueName: \"kubernetes.io/projected/90363ad8-673b-45b6-887f-c82f70632eba-kube-api-access-2dbc9\") pod \"limitador-limitador-85d9db4d5d-h7vrb\" (UID: \"90363ad8-673b-45b6-887f-c82f70632eba\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb" Jul 31 18:51:51.968576 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:51.968554 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/90363ad8-673b-45b6-887f-c82f70632eba-config-file\") pod \"limitador-limitador-85d9db4d5d-h7vrb\" (UID: \"90363ad8-673b-45b6-887f-c82f70632eba\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb" Jul 31 18:51:51.977889 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:51.977868 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dbc9\" (UniqueName: \"kubernetes.io/projected/90363ad8-673b-45b6-887f-c82f70632eba-kube-api-access-2dbc9\") pod \"limitador-limitador-85d9db4d5d-h7vrb\" (UID: \"90363ad8-673b-45b6-887f-c82f70632eba\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb" Jul 31 18:51:52.082886 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:52.082814 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb" Jul 31 18:51:52.222040 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:52.222008 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 18:51:52.224230 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:51:52.224201 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90363ad8_673b_45b6_887f_c82f70632eba.slice/crio-1be03204940b7909b509dbc4b2e8aaad4cd6615d920c5370f57c0ffe488beecf WatchSource:0}: Error finding container 1be03204940b7909b509dbc4b2e8aaad4cd6615d920c5370f57c0ffe488beecf: Status 404 returned error can't find the container with id 1be03204940b7909b509dbc4b2e8aaad4cd6615d920c5370f57c0ffe488beecf Jul 31 18:51:52.259972 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:52.259939 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:51:52.356513 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:52.356440 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:51:52.394620 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:52.394591 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-fm8bq"] Jul 31 18:51:52.397691 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:52.397671 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-fm8bq" Jul 31 18:51:52.399475 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:52.399456 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-s7w92\"" Jul 31 18:51:52.404766 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:52.404746 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-fm8bq"] Jul 31 18:51:52.574182 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:52.574145 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsw24\" (UniqueName: \"kubernetes.io/projected/71fb4dcd-4ca4-465c-96a1-dea1f94a291c-kube-api-access-zsw24\") pod \"authorino-c548b5d7d-fm8bq\" (UID: \"71fb4dcd-4ca4-465c-96a1-dea1f94a291c\") " pod="kuadrant-system/authorino-c548b5d7d-fm8bq" Jul 31 18:51:52.674708 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:52.674622 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsw24\" (UniqueName: \"kubernetes.io/projected/71fb4dcd-4ca4-465c-96a1-dea1f94a291c-kube-api-access-zsw24\") pod \"authorino-c548b5d7d-fm8bq\" (UID: \"71fb4dcd-4ca4-465c-96a1-dea1f94a291c\") " pod="kuadrant-system/authorino-c548b5d7d-fm8bq" Jul 31 18:51:52.685251 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:52.685215 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsw24\" (UniqueName: \"kubernetes.io/projected/71fb4dcd-4ca4-465c-96a1-dea1f94a291c-kube-api-access-zsw24\") pod \"authorino-c548b5d7d-fm8bq\" (UID: \"71fb4dcd-4ca4-465c-96a1-dea1f94a291c\") " pod="kuadrant-system/authorino-c548b5d7d-fm8bq" Jul 31 18:51:52.709108 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:52.709083 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-fm8bq" Jul 31 18:51:52.835261 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:52.835216 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-fm8bq"] Jul 31 18:51:52.838553 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:51:52.838518 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71fb4dcd_4ca4_465c_96a1_dea1f94a291c.slice/crio-bbdb42fbf7f14046fbe5691fd20ac5cd7b5cfb98a7a0b3849a36e8744810a415 WatchSource:0}: Error finding container bbdb42fbf7f14046fbe5691fd20ac5cd7b5cfb98a7a0b3849a36e8744810a415: Status 404 returned error can't find the container with id bbdb42fbf7f14046fbe5691fd20ac5cd7b5cfb98a7a0b3849a36e8744810a415 Jul 31 18:51:53.089810 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:53.089768 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb" event={"ID":"90363ad8-673b-45b6-887f-c82f70632eba","Type":"ContainerStarted","Data":"1be03204940b7909b509dbc4b2e8aaad4cd6615d920c5370f57c0ffe488beecf"} Jul 31 18:51:53.091193 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:53.091162 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-fm8bq" event={"ID":"71fb4dcd-4ca4-465c-96a1-dea1f94a291c","Type":"ContainerStarted","Data":"bbdb42fbf7f14046fbe5691fd20ac5cd7b5cfb98a7a0b3849a36e8744810a415"} Jul 31 18:51:54.097920 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:54.097879 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb" event={"ID":"90363ad8-673b-45b6-887f-c82f70632eba","Type":"ContainerStarted","Data":"16a490a4e5ff2e3eaa7ea01a47cdb8edff9d43f6ec0177125207bc2bac6bfd13"} Jul 31 18:51:54.098384 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:54.098032 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb" Jul 31 18:51:54.116677 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:54.116624 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb" podStartSLOduration=1.954595605 podStartE2EDuration="3.116605261s" totalImagesPullingTime="1.162009656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:51:54.11517672 +0000 UTC m=+490.753166521" watchObservedRunningTime="2026-07-31 18:51:54.116605261 +0000 UTC m=+490.754595062" Jul 31 18:51:55.103873 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:55.103833 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-fm8bq" event={"ID":"71fb4dcd-4ca4-465c-96a1-dea1f94a291c","Type":"ContainerStarted","Data":"06152a476210eda4f2873a54e9dd80a3bfd22c0b22e58d902c3ab61ec2bc2c9b"} Jul 31 18:51:55.121774 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:51:55.121729 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-fm8bq" podStartSLOduration=1.371434773 podStartE2EDuration="3.121714294s" totalImagesPullingTime="1.750279521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:51:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:51:55.120133183 +0000 UTC m=+491.758122984" watchObservedRunningTime="2026-07-31 18:51:55.121714294 +0000 UTC m=+491.759704093" Jul 31 18:52:05.105226 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:05.105196 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb" Jul 31 18:52:06.454199 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:06.454168 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:52:06.552852 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:06.552814 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:52:06.954601 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:06.954569 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:52:19.684577 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:19.684540 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:52:20.195583 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:20.195550 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:52:21.433763 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:21.433729 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-fm8bq"] Jul 31 18:52:21.434186 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:21.433900 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-fm8bq" podUID="71fb4dcd-4ca4-465c-96a1-dea1f94a291c" containerName="authorino" containerID="cri-o://06152a476210eda4f2873a54e9dd80a3bfd22c0b22e58d902c3ab61ec2bc2c9b" gracePeriod=30 Jul 31 18:52:21.683029 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:21.683003 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-fm8bq" Jul 31 18:52:21.703024 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:21.702965 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsw24\" (UniqueName: \"kubernetes.io/projected/71fb4dcd-4ca4-465c-96a1-dea1f94a291c-kube-api-access-zsw24\") pod \"71fb4dcd-4ca4-465c-96a1-dea1f94a291c\" (UID: \"71fb4dcd-4ca4-465c-96a1-dea1f94a291c\") " Jul 31 18:52:21.710354 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:21.706138 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71fb4dcd-4ca4-465c-96a1-dea1f94a291c-kube-api-access-zsw24" (OuterVolumeSpecName: "kube-api-access-zsw24") pod "71fb4dcd-4ca4-465c-96a1-dea1f94a291c" (UID: "71fb4dcd-4ca4-465c-96a1-dea1f94a291c"). InnerVolumeSpecName "kube-api-access-zsw24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 18:52:21.804447 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:21.804417 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsw24\" (UniqueName: \"kubernetes.io/projected/71fb4dcd-4ca4-465c-96a1-dea1f94a291c-kube-api-access-zsw24\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 18:52:22.231296 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:22.231263 2565 generic.go:356] "Generic (PLEG): container finished" podID="71fb4dcd-4ca4-465c-96a1-dea1f94a291c" containerID="06152a476210eda4f2873a54e9dd80a3bfd22c0b22e58d902c3ab61ec2bc2c9b" exitCode=0 Jul 31 18:52:22.231464 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:22.231312 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-fm8bq" Jul 31 18:52:22.231464 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:22.231323 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-fm8bq" event={"ID":"71fb4dcd-4ca4-465c-96a1-dea1f94a291c","Type":"ContainerDied","Data":"06152a476210eda4f2873a54e9dd80a3bfd22c0b22e58d902c3ab61ec2bc2c9b"} Jul 31 18:52:22.231464 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:22.231354 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-fm8bq" event={"ID":"71fb4dcd-4ca4-465c-96a1-dea1f94a291c","Type":"ContainerDied","Data":"bbdb42fbf7f14046fbe5691fd20ac5cd7b5cfb98a7a0b3849a36e8744810a415"} Jul 31 18:52:22.231464 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:22.231370 2565 scope.go:117] "RemoveContainer" containerID="06152a476210eda4f2873a54e9dd80a3bfd22c0b22e58d902c3ab61ec2bc2c9b" Jul 31 18:52:22.240152 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:22.240131 2565 scope.go:117] "RemoveContainer" containerID="06152a476210eda4f2873a54e9dd80a3bfd22c0b22e58d902c3ab61ec2bc2c9b" Jul 31 18:52:22.240451 ip-10-0-139-39 kubenswrapper[2565]: E0731 18:52:22.240431 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06152a476210eda4f2873a54e9dd80a3bfd22c0b22e58d902c3ab61ec2bc2c9b\": container with ID starting with 06152a476210eda4f2873a54e9dd80a3bfd22c0b22e58d902c3ab61ec2bc2c9b not found: ID does not exist" containerID="06152a476210eda4f2873a54e9dd80a3bfd22c0b22e58d902c3ab61ec2bc2c9b" Jul 31 18:52:22.240514 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:22.240459 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06152a476210eda4f2873a54e9dd80a3bfd22c0b22e58d902c3ab61ec2bc2c9b"} err="failed to get container status \"06152a476210eda4f2873a54e9dd80a3bfd22c0b22e58d902c3ab61ec2bc2c9b\": rpc error: code = NotFound desc = could not find container \"06152a476210eda4f2873a54e9dd80a3bfd22c0b22e58d902c3ab61ec2bc2c9b\": container with ID starting with 06152a476210eda4f2873a54e9dd80a3bfd22c0b22e58d902c3ab61ec2bc2c9b not found: ID does not exist" Jul 31 18:52:22.249414 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:22.249391 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-fm8bq"] Jul 31 18:52:22.254884 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:22.254865 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-fm8bq"] Jul 31 18:52:23.886432 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:23.886400 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71fb4dcd-4ca4-465c-96a1-dea1f94a291c" path="/var/lib/kubelet/pods/71fb4dcd-4ca4-465c-96a1-dea1f94a291c/volumes" Jul 31 18:52:26.050228 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:26.050188 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:52:26.211149 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:26.211097 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:52:26.282481 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:26.282443 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:52:26.356632 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:26.356549 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:52:29.354139 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:29.354107 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:52:29.455541 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:29.455509 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:52:31.083226 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:31.083188 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:52:32.042112 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:32.042074 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:52:56.690016 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:56.689983 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:52:56.792075 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:56.792038 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:52:58.206584 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:58.206545 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:52:58.464452 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:58.464350 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:52:58.886302 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:58.886273 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:52:58.961985 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:58.961944 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:52:59.184905 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:52:59.184827 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 18:53:08.461685 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:08.461582 2565 generic.go:356] "Generic (PLEG): container finished" podID="ed4e23e4-7b4d-4214-a571-0f54d841c7c5" containerID="76263e46c83ba6db6bf7c26521b6a601eff9d3d6ddd4efce1b15c153e04f5bbd" exitCode=0 Jul 31 18:53:08.461685 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:08.461656 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" event={"ID":"ed4e23e4-7b4d-4214-a571-0f54d841c7c5","Type":"ContainerDied","Data":"76263e46c83ba6db6bf7c26521b6a601eff9d3d6ddd4efce1b15c153e04f5bbd"} Jul 31 18:53:08.462127 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:08.462107 2565 scope.go:117] "RemoveContainer" containerID="76263e46c83ba6db6bf7c26521b6a601eff9d3d6ddd4efce1b15c153e04f5bbd" Jul 31 18:53:08.463261 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:08.463211 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 18:53:08.463333 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:08.463307 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 18:53:08.463370 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:08.463349 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 18:53:08.602583 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:08.602551 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:53:08.712848 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:08.712761 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:53:09.468576 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:09.468541 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" event={"ID":"ed4e23e4-7b4d-4214-a571-0f54d841c7c5","Type":"ContainerStarted","Data":"d294dd70015e13ac712bf75c5bd1a6d0c7e7e57a6be5e4aa81566689c1eaddfb"} Jul 31 18:53:09.681889 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:09.681843 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:53:09.682088 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:09.682024 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-w8k4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 31 18:53:09.682088 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:09.682065 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" podUID="ed4e23e4-7b4d-4214-a571-0f54d841c7c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 31 18:53:10.681968 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:10.681928 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-w8k4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 31 18:53:10.682389 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:10.681986 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" podUID="ed4e23e4-7b4d-4214-a571-0f54d841c7c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 31 18:53:11.681824 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:11.681783 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-w8k4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 31 18:53:11.681999 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:11.681843 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" podUID="ed4e23e4-7b4d-4214-a571-0f54d841c7c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 31 18:53:12.682259 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:12.682195 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-w8k4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 31 18:53:12.682714 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:12.682288 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" podUID="ed4e23e4-7b4d-4214-a571-0f54d841c7c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 31 18:53:13.682479 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:13.682433 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-w8k4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 31 18:53:13.682847 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:13.682498 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" podUID="ed4e23e4-7b4d-4214-a571-0f54d841c7c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 31 18:53:14.715682 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:14.715638 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:53:14.716208 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:14.716185 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:53:14.717003 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:14.716982 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w8k4k" Jul 31 18:53:17.724773 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:17.724728 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:53:18.121088 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:18.121047 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 18:53:22.513684 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:22.513645 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:53:23.286471 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:23.286441 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 18:53:25.171156 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:25.171120 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:53:25.271876 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:25.271834 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:53:30.511211 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:30.511165 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:53:31.488120 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:31.488082 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 18:53:39.590447 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:39.590408 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4"] Jul 31 18:53:39.590817 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:39.590762 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71fb4dcd-4ca4-465c-96a1-dea1f94a291c" containerName="authorino" Jul 31 18:53:39.590817 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:39.590773 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="71fb4dcd-4ca4-465c-96a1-dea1f94a291c" containerName="authorino" Jul 31 18:53:39.590890 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:39.590844 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="71fb4dcd-4ca4-465c-96a1-dea1f94a291c" containerName="authorino" Jul 31 18:53:39.594824 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:39.594804 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4" Jul 31 18:53:39.597451 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:39.597427 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bs6tj\"" Jul 31 18:53:39.597451 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:39.597449 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 31 18:53:39.597627 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:39.597455 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 31 18:53:39.597627 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:39.597481 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 31 18:53:39.604630 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:39.604608 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4"] Jul 31 18:53:39.698601 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:39.698563 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:53:39.706788 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:39.706760 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10caf125-874d-4caf-bbfc-0d196da1008b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4\" (UID: \"10caf125-874d-4caf-bbfc-0d196da1008b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4" Jul 31 18:53:39.706915 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:39.706819 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10caf125-874d-4caf-bbfc-0d196da1008b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4\" (UID: \"10caf125-874d-4caf-bbfc-0d196da1008b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4" Jul 31 18:53:39.706915 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:39.706854 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-946lw\" (UniqueName: \"kubernetes.io/projected/10caf125-874d-4caf-bbfc-0d196da1008b-kube-api-access-946lw\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4\" (UID: \"10caf125-874d-4caf-bbfc-0d196da1008b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4" Jul 31 18:53:39.706915 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:39.706889 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10caf125-874d-4caf-bbfc-0d196da1008b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4\" (UID: \"10caf125-874d-4caf-bbfc-0d196da1008b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4" Jul 31 18:53:39.707042 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:39.706920 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10caf125-874d-4caf-bbfc-0d196da1008b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4\" (UID: \"10caf125-874d-4caf-bbfc-0d196da1008b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4" Jul 31 18:53:39.707042 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:39.706951 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10caf125-874d-4caf-bbfc-0d196da1008b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4\" (UID: \"10caf125-874d-4caf-bbfc-0d196da1008b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4" Jul 31 18:53:39.807858 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:39.807821 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10caf125-874d-4caf-bbfc-0d196da1008b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4\" (UID: \"10caf125-874d-4caf-bbfc-0d196da1008b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4" Jul 31 18:53:39.807858 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:39.807858 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-946lw\" (UniqueName: \"kubernetes.io/projected/10caf125-874d-4caf-bbfc-0d196da1008b-kube-api-access-946lw\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4\" (UID: \"10caf125-874d-4caf-bbfc-0d196da1008b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4" Jul 31 18:53:39.808126 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:39.807889 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10caf125-874d-4caf-bbfc-0d196da1008b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4\" (UID: \"10caf125-874d-4caf-bbfc-0d196da1008b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4" Jul 31 18:53:39.808126 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:39.807922 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10caf125-874d-4caf-bbfc-0d196da1008b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4\" (UID: \"10caf125-874d-4caf-bbfc-0d196da1008b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4" Jul 31 18:53:39.808126 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:39.808060 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10caf125-874d-4caf-bbfc-0d196da1008b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4\" (UID: \"10caf125-874d-4caf-bbfc-0d196da1008b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4" Jul 31 18:53:39.808350 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:39.808308 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10caf125-874d-4caf-bbfc-0d196da1008b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4\" (UID: \"10caf125-874d-4caf-bbfc-0d196da1008b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4" Jul 31 18:53:39.808415 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:39.808398 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10caf125-874d-4caf-bbfc-0d196da1008b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4\" (UID: \"10caf125-874d-4caf-bbfc-0d196da1008b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4" Jul 31 18:53:39.808509 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:39.808485 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10caf125-874d-4caf-bbfc-0d196da1008b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4\" (UID: \"10caf125-874d-4caf-bbfc-0d196da1008b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4" Jul 31 18:53:39.808672 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:39.808643 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10caf125-874d-4caf-bbfc-0d196da1008b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4\" (UID: \"10caf125-874d-4caf-bbfc-0d196da1008b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4" Jul 31 18:53:39.810188 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:39.810159 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10caf125-874d-4caf-bbfc-0d196da1008b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4\" (UID: \"10caf125-874d-4caf-bbfc-0d196da1008b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4" Jul 31 18:53:39.810596 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:39.810574 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10caf125-874d-4caf-bbfc-0d196da1008b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4\" (UID: \"10caf125-874d-4caf-bbfc-0d196da1008b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4" Jul 31 18:53:39.817812 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:39.817779 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-946lw\" (UniqueName: \"kubernetes.io/projected/10caf125-874d-4caf-bbfc-0d196da1008b-kube-api-access-946lw\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4\" (UID: \"10caf125-874d-4caf-bbfc-0d196da1008b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4" Jul 31 18:53:39.906114 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:39.906025 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4" Jul 31 18:53:40.048702 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:40.048677 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4"] Jul 31 18:53:40.050782 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:53:40.050751 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10caf125_874d_4caf_bbfc_0d196da1008b.slice/crio-28251dfcfd361f36cded90fe6bea3c095b83073d9fe38e84099888840a2c36f0 WatchSource:0}: Error finding container 28251dfcfd361f36cded90fe6bea3c095b83073d9fe38e84099888840a2c36f0: Status 404 returned error can't find the container with id 28251dfcfd361f36cded90fe6bea3c095b83073d9fe38e84099888840a2c36f0 Jul 31 18:53:40.634581 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:40.634510 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4" event={"ID":"10caf125-874d-4caf-bbfc-0d196da1008b","Type":"ContainerStarted","Data":"28251dfcfd361f36cded90fe6bea3c095b83073d9fe38e84099888840a2c36f0"} Jul 31 18:53:41.106272 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:41.106227 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 18:53:41.641921 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:41.641870 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4" event={"ID":"10caf125-874d-4caf-bbfc-0d196da1008b","Type":"ContainerStarted","Data":"fe6641569f657dd287b455735ac4a3974642c890e4563d73e5b936f8e85ca87f"} Jul 31 18:53:41.660989 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:41.660933 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4" podStartSLOduration=1.63792512 podStartE2EDuration="2.660914961s" totalImagesPullingTime="1.022989841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:53:41.65892773 +0000 UTC m=+598.296917522" watchObservedRunningTime="2026-07-31 18:53:41.660914961 +0000 UTC m=+598.298904764" Jul 31 18:53:49.907042 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:49.907006 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4" Jul 31 18:53:49.907438 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:49.907079 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4" Jul 31 18:53:49.919386 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:49.919359 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4" Jul 31 18:53:50.695940 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:50.695907 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4" Jul 31 18:53:56.540765 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:53:56.540731 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:54:10.684463 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:10.684384 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:54:12.785104 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:12.785067 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 18:54:18.636386 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:18.636345 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv"] Jul 31 18:54:18.645346 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:18.645312 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv" Jul 31 18:54:18.647783 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:18.647450 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 31 18:54:18.650421 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:18.650351 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:54:18.650872 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:18.650852 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv"] Jul 31 18:54:18.760030 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:18.759996 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc47d375-9655-47db-8feb-e96c91236d91-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv\" (UID: \"cc47d375-9655-47db-8feb-e96c91236d91\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv" Jul 31 18:54:18.760218 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:18.760039 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc47d375-9655-47db-8feb-e96c91236d91-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv\" (UID: \"cc47d375-9655-47db-8feb-e96c91236d91\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv" Jul 31 18:54:18.760218 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:18.760097 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc47d375-9655-47db-8feb-e96c91236d91-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv\" (UID: \"cc47d375-9655-47db-8feb-e96c91236d91\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv" Jul 31 18:54:18.760218 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:18.760137 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc47d375-9655-47db-8feb-e96c91236d91-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv\" (UID: \"cc47d375-9655-47db-8feb-e96c91236d91\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv" Jul 31 18:54:18.760218 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:18.760161 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc47d375-9655-47db-8feb-e96c91236d91-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv\" (UID: \"cc47d375-9655-47db-8feb-e96c91236d91\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv" Jul 31 18:54:18.760433 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:18.760294 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x84v\" (UniqueName: \"kubernetes.io/projected/cc47d375-9655-47db-8feb-e96c91236d91-kube-api-access-9x84v\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv\" (UID: \"cc47d375-9655-47db-8feb-e96c91236d91\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv" Jul 31 18:54:18.861042 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:18.861008 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc47d375-9655-47db-8feb-e96c91236d91-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv\" (UID: \"cc47d375-9655-47db-8feb-e96c91236d91\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv" Jul 31 18:54:18.861272 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:18.861048 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc47d375-9655-47db-8feb-e96c91236d91-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv\" (UID: \"cc47d375-9655-47db-8feb-e96c91236d91\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv" Jul 31 18:54:18.861272 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:18.861073 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc47d375-9655-47db-8feb-e96c91236d91-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv\" (UID: \"cc47d375-9655-47db-8feb-e96c91236d91\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv" Jul 31 18:54:18.861272 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:18.861147 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x84v\" (UniqueName: \"kubernetes.io/projected/cc47d375-9655-47db-8feb-e96c91236d91-kube-api-access-9x84v\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv\" (UID: \"cc47d375-9655-47db-8feb-e96c91236d91\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv" Jul 31 18:54:18.861272 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:18.861203 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc47d375-9655-47db-8feb-e96c91236d91-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv\" (UID: \"cc47d375-9655-47db-8feb-e96c91236d91\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv" Jul 31 18:54:18.861272 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:18.861226 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc47d375-9655-47db-8feb-e96c91236d91-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv\" (UID: \"cc47d375-9655-47db-8feb-e96c91236d91\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv" Jul 31 18:54:18.861646 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:18.861515 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc47d375-9655-47db-8feb-e96c91236d91-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv\" (UID: \"cc47d375-9655-47db-8feb-e96c91236d91\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv" Jul 31 18:54:18.861646 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:18.861556 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc47d375-9655-47db-8feb-e96c91236d91-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv\" (UID: \"cc47d375-9655-47db-8feb-e96c91236d91\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv" Jul 31 18:54:18.861646 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:18.861577 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc47d375-9655-47db-8feb-e96c91236d91-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv\" (UID: \"cc47d375-9655-47db-8feb-e96c91236d91\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv" Jul 31 18:54:18.863590 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:18.863569 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc47d375-9655-47db-8feb-e96c91236d91-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv\" (UID: \"cc47d375-9655-47db-8feb-e96c91236d91\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv" Jul 31 18:54:18.863754 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:18.863738 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc47d375-9655-47db-8feb-e96c91236d91-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv\" (UID: \"cc47d375-9655-47db-8feb-e96c91236d91\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv" Jul 31 18:54:18.869438 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:18.869414 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x84v\" (UniqueName: \"kubernetes.io/projected/cc47d375-9655-47db-8feb-e96c91236d91-kube-api-access-9x84v\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv\" (UID: \"cc47d375-9655-47db-8feb-e96c91236d91\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv" Jul 31 18:54:18.956879 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:18.956800 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv" Jul 31 18:54:19.097270 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:19.097230 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv"] Jul 31 18:54:19.099427 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:54:19.099403 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc47d375_9655_47db_8feb_e96c91236d91.slice/crio-d21745f2de8a1953a51684ca8805908c6940b4e1a8f3a232af3b1f9cdee8f1c3 WatchSource:0}: Error finding container d21745f2de8a1953a51684ca8805908c6940b4e1a8f3a232af3b1f9cdee8f1c3: Status 404 returned error can't find the container with id d21745f2de8a1953a51684ca8805908c6940b4e1a8f3a232af3b1f9cdee8f1c3 Jul 31 18:54:19.100818 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:19.100798 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 18:54:19.823963 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:19.823932 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv" event={"ID":"cc47d375-9655-47db-8feb-e96c91236d91","Type":"ContainerStarted","Data":"93324fcab873b60c750d97458192f6cc71d877c4067d032889d89a1838c490b4"} Jul 31 18:54:19.824481 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:19.823977 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv" event={"ID":"cc47d375-9655-47db-8feb-e96c91236d91","Type":"ContainerStarted","Data":"d21745f2de8a1953a51684ca8805908c6940b4e1a8f3a232af3b1f9cdee8f1c3"} Jul 31 18:54:19.843709 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:19.843654 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv" podStartSLOduration=1.6697031500000001 podStartE2EDuration="1.843638106s" totalImagesPullingTime="173.934956ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:54:19.841448497 +0000 UTC m=+636.479438297" watchObservedRunningTime="2026-07-31 18:54:19.843638106 +0000 UTC m=+636.481627906" Jul 31 18:54:21.296650 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:21.296617 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 18:54:28.957095 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:28.957056 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv" Jul 31 18:54:28.957493 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:28.957117 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv" Jul 31 18:54:28.969629 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:28.969599 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv" Jul 31 18:54:29.890270 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:29.890228 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv" Jul 31 18:54:59.079489 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:54:59.079450 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:55:27.898115 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:55:27.898075 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-9bd55985b-2p6jj"] Jul 31 18:55:27.901514 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:55:27.901496 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9bd55985b-2p6jj" Jul 31 18:55:27.903831 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:55:27.903805 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-s7w92\"" Jul 31 18:55:27.903940 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:55:27.903845 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 31 18:55:27.914151 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:55:27.914130 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9bd55985b-2p6jj"] Jul 31 18:55:28.061839 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:55:28.061800 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8cwr\" (UniqueName: \"kubernetes.io/projected/8b24f01a-7f75-43e1-9a87-631a7a69eca8-kube-api-access-p8cwr\") pod \"authorino-9bd55985b-2p6jj\" (UID: \"8b24f01a-7f75-43e1-9a87-631a7a69eca8\") " pod="kuadrant-system/authorino-9bd55985b-2p6jj" Jul 31 18:55:28.062012 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:55:28.061877 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8b24f01a-7f75-43e1-9a87-631a7a69eca8-tls-cert\") pod \"authorino-9bd55985b-2p6jj\" (UID: \"8b24f01a-7f75-43e1-9a87-631a7a69eca8\") " pod="kuadrant-system/authorino-9bd55985b-2p6jj" Jul 31 18:55:28.162761 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:55:28.162674 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8cwr\" (UniqueName: \"kubernetes.io/projected/8b24f01a-7f75-43e1-9a87-631a7a69eca8-kube-api-access-p8cwr\") pod \"authorino-9bd55985b-2p6jj\" (UID: \"8b24f01a-7f75-43e1-9a87-631a7a69eca8\") " pod="kuadrant-system/authorino-9bd55985b-2p6jj" Jul 31 18:55:28.162761 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:55:28.162753 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8b24f01a-7f75-43e1-9a87-631a7a69eca8-tls-cert\") pod \"authorino-9bd55985b-2p6jj\" (UID: \"8b24f01a-7f75-43e1-9a87-631a7a69eca8\") " pod="kuadrant-system/authorino-9bd55985b-2p6jj" Jul 31 18:55:28.165185 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:55:28.165159 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8b24f01a-7f75-43e1-9a87-631a7a69eca8-tls-cert\") pod \"authorino-9bd55985b-2p6jj\" (UID: \"8b24f01a-7f75-43e1-9a87-631a7a69eca8\") " pod="kuadrant-system/authorino-9bd55985b-2p6jj" Jul 31 18:55:28.172096 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:55:28.172071 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8cwr\" (UniqueName: \"kubernetes.io/projected/8b24f01a-7f75-43e1-9a87-631a7a69eca8-kube-api-access-p8cwr\") pod \"authorino-9bd55985b-2p6jj\" (UID: \"8b24f01a-7f75-43e1-9a87-631a7a69eca8\") " pod="kuadrant-system/authorino-9bd55985b-2p6jj" Jul 31 18:55:28.209852 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:55:28.209822 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9bd55985b-2p6jj" Jul 31 18:55:28.342821 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:55:28.342792 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9bd55985b-2p6jj"] Jul 31 18:55:28.344201 ip-10-0-139-39 kubenswrapper[2565]: W0731 18:55:28.344179 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b24f01a_7f75_43e1_9a87_631a7a69eca8.slice/crio-eff58d2f983e5b225e3f430b0a7e80bced5d7e846616c7651df20b28e210b3cc WatchSource:0}: Error finding container eff58d2f983e5b225e3f430b0a7e80bced5d7e846616c7651df20b28e210b3cc: Status 404 returned error can't find the container with id eff58d2f983e5b225e3f430b0a7e80bced5d7e846616c7651df20b28e210b3cc Jul 31 18:55:28.969689 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:55:28.969651 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:55:29.070982 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:55:29.070949 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:55:29.166054 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:55:29.166026 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9bd55985b-2p6jj" event={"ID":"8b24f01a-7f75-43e1-9a87-631a7a69eca8","Type":"ContainerStarted","Data":"eff58d2f983e5b225e3f430b0a7e80bced5d7e846616c7651df20b28e210b3cc"} Jul 31 18:55:30.172615 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:55:30.172573 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9bd55985b-2p6jj" event={"ID":"8b24f01a-7f75-43e1-9a87-631a7a69eca8","Type":"ContainerStarted","Data":"b1f9bca03083ee1dcb362bf4232aa902b72b5a27dac08c9b00b04875cd2c3552"} Jul 31 18:55:30.192172 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:55:30.192105 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-9bd55985b-2p6jj" podStartSLOduration=2.428359875 podStartE2EDuration="3.192086017s" totalImagesPullingTime="763.726142ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:55:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:55:30.189876136 +0000 UTC m=+706.827865938" watchObservedRunningTime="2026-07-31 18:55:30.192086017 +0000 UTC m=+706.830075817" Jul 31 18:55:30.723526 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:55:30.723481 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:55:30.818218 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:55:30.818181 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:55:31.801558 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:55:31.801529 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:55:31.897814 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:55:31.897779 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:55:59.392508 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:55:59.392433 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 18:56:08.288598 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:56:08.288558 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 18:56:17.368889 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:56:17.368850 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:56:17.466344 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:56:17.466312 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:56:21.497789 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:56:21.497758 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 18:56:43.574183 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:56:43.574145 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:56:43.673975 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:56:43.673943 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:57:07.619735 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:57:07.619661 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:57:36.407974 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:57:36.407935 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:57:37.734075 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:57:37.734036 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:58:00.592657 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:58:00.592616 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 18:58:09.890517 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:58:09.890486 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 18:58:40.241807 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:58:40.241706 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:58:40.378720 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:58:40.378678 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:58:40.539313 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:58:40.539276 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:58:46.672686 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:58:46.672641 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:58:46.739357 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:58:46.739323 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:58:46.816196 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:58:46.816160 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:58:47.002075 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:58:47.002018 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:58:53.125210 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:58:53.125169 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:58:53.192121 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:58:53.192088 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 18:59:08.806038 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:59:08.806006 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 18:59:16.086562 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:59:16.086523 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 18:59:31.690485 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:59:31.690447 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 18:59:37.486521 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:59:37.486487 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 18:59:46.892835 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:59:46.892794 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 18:59:53.294108 ip-10-0-139-39 kubenswrapper[2565]: I0731 18:59:53.294070 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:00:07.408934 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:00:07.408820 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:00:12.595135 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:00:12.595101 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:00:17.590100 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:00:17.590062 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:00:39.394347 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:00:39.394309 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:00:50.203830 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:00:50.203792 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:00:55.889099 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:00:55.889066 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:01:12.799259 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:01:12.799187 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:01:22.994983 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:01:22.994946 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:01:28.906376 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:01:28.906336 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:01:36.091666 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:01:36.091624 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:01:45.102876 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:01:45.102836 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:01:49.395859 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:01:49.395823 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:01:54.587165 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:01:54.587128 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:02:01.301645 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:02:01.301604 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:02:13.193273 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:02:13.193218 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:02:20.194365 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:02:20.194325 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:02:31.295504 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:02:31.295465 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:02:36.636605 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:02:36.636566 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:02:46.282566 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:02:46.282530 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:02:52.095727 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:02:52.095687 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:02:57.093950 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:02:57.093909 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:02:59.854275 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:02:59.854217 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:02:59.921538 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:02:59.921508 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:03:02.529751 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:03:02.529718 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:03:02.631370 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:03:02.631336 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:03:07.392326 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:03:07.392221 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:04:02.305352 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:04:02.305318 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bftdk"] Jul 31 19:04:02.764085 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:04:02.764044 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bftdk" podUID="ec504d44-ecbb-41dc-9aa6-34e58c000207" containerName="manager" containerID="cri-o://e639fea3b10027ee73576d4489e61030a6c58a6732276b6497ca28dd82b6cdeb" gracePeriod=10 Jul 31 19:04:03.024452 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:04:03.024388 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bftdk" Jul 31 19:04:03.124336 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:04:03.124303 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec504d44-ecbb-41dc-9aa6-34e58c000207-extensions-socket-volume\") pod \"ec504d44-ecbb-41dc-9aa6-34e58c000207\" (UID: \"ec504d44-ecbb-41dc-9aa6-34e58c000207\") " Jul 31 19:04:03.124517 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:04:03.124363 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ftlk\" (UniqueName: \"kubernetes.io/projected/ec504d44-ecbb-41dc-9aa6-34e58c000207-kube-api-access-7ftlk\") pod \"ec504d44-ecbb-41dc-9aa6-34e58c000207\" (UID: \"ec504d44-ecbb-41dc-9aa6-34e58c000207\") " Jul 31 19:04:03.124709 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:04:03.124684 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec504d44-ecbb-41dc-9aa6-34e58c000207-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ec504d44-ecbb-41dc-9aa6-34e58c000207" (UID: "ec504d44-ecbb-41dc-9aa6-34e58c000207"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:04:03.126401 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:04:03.126374 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec504d44-ecbb-41dc-9aa6-34e58c000207-kube-api-access-7ftlk" (OuterVolumeSpecName: "kube-api-access-7ftlk") pod "ec504d44-ecbb-41dc-9aa6-34e58c000207" (UID: "ec504d44-ecbb-41dc-9aa6-34e58c000207"). InnerVolumeSpecName "kube-api-access-7ftlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:04:03.225643 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:04:03.225610 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec504d44-ecbb-41dc-9aa6-34e58c000207-extensions-socket-volume\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:04:03.225643 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:04:03.225638 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ftlk\" (UniqueName: \"kubernetes.io/projected/ec504d44-ecbb-41dc-9aa6-34e58c000207-kube-api-access-7ftlk\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:04:03.769507 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:04:03.769467 2565 generic.go:356] "Generic (PLEG): container finished" podID="ec504d44-ecbb-41dc-9aa6-34e58c000207" containerID="e639fea3b10027ee73576d4489e61030a6c58a6732276b6497ca28dd82b6cdeb" exitCode=0 Jul 31 19:04:03.769507 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:04:03.769507 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bftdk" Jul 31 19:04:03.769950 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:04:03.769535 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bftdk" event={"ID":"ec504d44-ecbb-41dc-9aa6-34e58c000207","Type":"ContainerDied","Data":"e639fea3b10027ee73576d4489e61030a6c58a6732276b6497ca28dd82b6cdeb"} Jul 31 19:04:03.769950 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:04:03.769579 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bftdk" event={"ID":"ec504d44-ecbb-41dc-9aa6-34e58c000207","Type":"ContainerDied","Data":"a3953ba4ab47b0ce888d07d914f6f8cec81d4b45961d4f53fadc8a0993f42d96"} Jul 31 19:04:03.769950 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:04:03.769603 2565 scope.go:117] "RemoveContainer" containerID="e639fea3b10027ee73576d4489e61030a6c58a6732276b6497ca28dd82b6cdeb" Jul 31 19:04:03.779389 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:04:03.779373 2565 scope.go:117] "RemoveContainer" containerID="e639fea3b10027ee73576d4489e61030a6c58a6732276b6497ca28dd82b6cdeb" Jul 31 19:04:03.779654 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:04:03.779635 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e639fea3b10027ee73576d4489e61030a6c58a6732276b6497ca28dd82b6cdeb\": container with ID starting with e639fea3b10027ee73576d4489e61030a6c58a6732276b6497ca28dd82b6cdeb not found: ID does not exist" containerID="e639fea3b10027ee73576d4489e61030a6c58a6732276b6497ca28dd82b6cdeb" Jul 31 19:04:03.779719 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:04:03.779666 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e639fea3b10027ee73576d4489e61030a6c58a6732276b6497ca28dd82b6cdeb"} err="failed to get container status \"e639fea3b10027ee73576d4489e61030a6c58a6732276b6497ca28dd82b6cdeb\": rpc error: code = NotFound desc = could not find container \"e639fea3b10027ee73576d4489e61030a6c58a6732276b6497ca28dd82b6cdeb\": container with ID starting with e639fea3b10027ee73576d4489e61030a6c58a6732276b6497ca28dd82b6cdeb not found: ID does not exist" Jul 31 19:04:03.792559 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:04:03.792536 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bftdk"] Jul 31 19:04:03.799129 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:04:03.799108 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-bftdk"] Jul 31 19:04:03.886913 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:04:03.886883 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec504d44-ecbb-41dc-9aa6-34e58c000207" path="/var/lib/kubelet/pods/ec504d44-ecbb-41dc-9aa6-34e58c000207/volumes" Jul 31 19:05:24.851586 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:05:24.851539 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:05:30.052804 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:05:30.052770 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:05:53.670003 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:05:53.669964 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:06:15.260466 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:06:15.260387 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:06:24.756389 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:06:24.756351 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:06:30.358768 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:06:30.358734 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:06:35.550593 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:06:35.550555 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:06:45.974798 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:06:45.974759 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:06:50.956637 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:06:50.956589 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:07:01.248308 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:07:01.248273 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:07:06.366403 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:07:06.366366 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:07:16.455900 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:07:16.455864 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:07:23.052475 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:07:23.052436 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:07:32.459700 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:07:32.459663 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:07:54.356601 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:07:54.356508 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:08:02.456378 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:02.456342 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:08:03.594642 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:03.594604 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd"] Jul 31 19:08:03.595151 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:03.595132 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec504d44-ecbb-41dc-9aa6-34e58c000207" containerName="manager" Jul 31 19:08:03.595273 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:03.595154 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec504d44-ecbb-41dc-9aa6-34e58c000207" containerName="manager" Jul 31 19:08:03.595412 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:03.595387 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec504d44-ecbb-41dc-9aa6-34e58c000207" containerName="manager" Jul 31 19:08:03.598257 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:03.598229 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd" Jul 31 19:08:03.602279 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:03.602257 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-f814dccf-kserve-self-signed-certs\"" Jul 31 19:08:03.626929 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:03.626906 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd"] Jul 31 19:08:03.654818 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:03.654794 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnrlq\" (UniqueName: \"kubernetes.io/projected/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-kube-api-access-vnrlq\") pod \"e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd\" (UID: \"7c3b548c-246e-4bdf-9a86-f318a77e1a7f\") " pod="llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd" Jul 31 19:08:03.654923 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:03.654836 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-home\") pod \"e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd\" (UID: \"7c3b548c-246e-4bdf-9a86-f318a77e1a7f\") " pod="llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd" Jul 31 19:08:03.654923 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:03.654858 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-dshm\") pod \"e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd\" (UID: \"7c3b548c-246e-4bdf-9a86-f318a77e1a7f\") " pod="llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd" Jul 31 19:08:03.654995 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:03.654916 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-model-cache\") pod \"e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd\" (UID: \"7c3b548c-246e-4bdf-9a86-f318a77e1a7f\") " pod="llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd" Jul 31 19:08:03.654995 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:03.654974 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-tmp-dir\") pod \"e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd\" (UID: \"7c3b548c-246e-4bdf-9a86-f318a77e1a7f\") " pod="llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd" Jul 31 19:08:03.655060 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:03.655014 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-tls-certs\") pod \"e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd\" (UID: \"7c3b548c-246e-4bdf-9a86-f318a77e1a7f\") " pod="llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd" Jul 31 19:08:03.730277 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:03.730223 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:08:03.756455 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:03.756423 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnrlq\" (UniqueName: \"kubernetes.io/projected/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-kube-api-access-vnrlq\") pod \"e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd\" (UID: \"7c3b548c-246e-4bdf-9a86-f318a77e1a7f\") " pod="llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd" Jul 31 19:08:03.756610 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:03.756481 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-home\") pod \"e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd\" (UID: \"7c3b548c-246e-4bdf-9a86-f318a77e1a7f\") " pod="llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd" Jul 31 19:08:03.756610 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:03.756509 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-dshm\") pod \"e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd\" (UID: \"7c3b548c-246e-4bdf-9a86-f318a77e1a7f\") " pod="llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd" Jul 31 19:08:03.756610 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:03.756535 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-model-cache\") pod \"e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd\" (UID: \"7c3b548c-246e-4bdf-9a86-f318a77e1a7f\") " pod="llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd" Jul 31 19:08:03.756610 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:03.756571 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-tmp-dir\") pod \"e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd\" (UID: \"7c3b548c-246e-4bdf-9a86-f318a77e1a7f\") " pod="llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd" Jul 31 19:08:03.756850 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:03.756616 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-tls-certs\") pod \"e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd\" (UID: \"7c3b548c-246e-4bdf-9a86-f318a77e1a7f\") " pod="llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd" Jul 31 19:08:03.756939 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:03.756918 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-home\") pod \"e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd\" (UID: \"7c3b548c-246e-4bdf-9a86-f318a77e1a7f\") " pod="llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd" Jul 31 19:08:03.757018 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:03.756996 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-model-cache\") pod \"e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd\" (UID: \"7c3b548c-246e-4bdf-9a86-f318a77e1a7f\") " pod="llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd" Jul 31 19:08:03.757081 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:03.757061 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-tmp-dir\") pod \"e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd\" (UID: \"7c3b548c-246e-4bdf-9a86-f318a77e1a7f\") " pod="llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd" Jul 31 19:08:03.759093 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:03.759072 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-dshm\") pod \"e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd\" (UID: \"7c3b548c-246e-4bdf-9a86-f318a77e1a7f\") " pod="llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd" Jul 31 19:08:03.759424 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:03.759402 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-tls-certs\") pod \"e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd\" (UID: \"7c3b548c-246e-4bdf-9a86-f318a77e1a7f\") " pod="llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd" Jul 31 19:08:03.766933 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:03.766907 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnrlq\" (UniqueName: \"kubernetes.io/projected/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-kube-api-access-vnrlq\") pod \"e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd\" (UID: \"7c3b548c-246e-4bdf-9a86-f318a77e1a7f\") " pod="llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd" Jul 31 19:08:03.907544 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:03.907462 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd" Jul 31 19:08:04.038996 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:04.038972 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd"] Jul 31 19:08:04.041071 ip-10-0-139-39 kubenswrapper[2565]: W0731 19:08:04.041045 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c3b548c_246e_4bdf_9a86_f318a77e1a7f.slice/crio-bc6be7839e74f1c777c6ab866fdb22967ec895e31e3bb96fdeeaedaa83a2426c WatchSource:0}: Error finding container bc6be7839e74f1c777c6ab866fdb22967ec895e31e3bb96fdeeaedaa83a2426c: Status 404 returned error can't find the container with id bc6be7839e74f1c777c6ab866fdb22967ec895e31e3bb96fdeeaedaa83a2426c Jul 31 19:08:04.955076 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:04.955035 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd" event={"ID":"7c3b548c-246e-4bdf-9a86-f318a77e1a7f","Type":"ContainerStarted","Data":"9b7302dd865848a6fd792ea0e835eb341e875a62485f8b984f56927b4c91b217"} Jul 31 19:08:04.955485 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:04.955085 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd" event={"ID":"7c3b548c-246e-4bdf-9a86-f318a77e1a7f","Type":"ContainerStarted","Data":"bc6be7839e74f1c777c6ab866fdb22967ec895e31e3bb96fdeeaedaa83a2426c"} Jul 31 19:08:04.972706 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:04.972657 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd" podStartSLOduration=1.972641973 podStartE2EDuration="1.972641973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:08:04.971917081 +0000 UTC m=+1461.609906880" watchObservedRunningTime="2026-07-31 19:08:04.972641973 +0000 UTC m=+1461.610631751" Jul 31 19:08:12.154901 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:12.154868 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:08:12.790310 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:12.790274 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r"] Jul 31 19:08:12.794084 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:12.794059 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r" Jul 31 19:08:12.795761 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:12.795741 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-f814dccf-kserve-self-signed-certs\"" Jul 31 19:08:12.804155 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:12.804136 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r"] Jul 31 19:08:12.825994 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:12.825959 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:08:12.947374 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:12.947333 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eae0070-4b15-449f-88df-f8fe2b3795dd-model-cache\") pod \"e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r\" (UID: \"1eae0070-4b15-449f-88df-f8fe2b3795dd\") " pod="llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r" Jul 31 19:08:12.947374 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:12.947372 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eae0070-4b15-449f-88df-f8fe2b3795dd-tmp-dir\") pod \"e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r\" (UID: \"1eae0070-4b15-449f-88df-f8fe2b3795dd\") " pod="llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r" Jul 31 19:08:12.947630 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:12.947393 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eae0070-4b15-449f-88df-f8fe2b3795dd-home\") pod \"e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r\" (UID: \"1eae0070-4b15-449f-88df-f8fe2b3795dd\") " pod="llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r" Jul 31 19:08:12.947630 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:12.947452 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-877nk\" (UniqueName: \"kubernetes.io/projected/1eae0070-4b15-449f-88df-f8fe2b3795dd-kube-api-access-877nk\") pod \"e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r\" (UID: \"1eae0070-4b15-449f-88df-f8fe2b3795dd\") " pod="llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r" Jul 31 19:08:12.947630 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:12.947489 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eae0070-4b15-449f-88df-f8fe2b3795dd-tls-certs\") pod \"e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r\" (UID: \"1eae0070-4b15-449f-88df-f8fe2b3795dd\") " pod="llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r" Jul 31 19:08:12.947630 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:12.947568 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eae0070-4b15-449f-88df-f8fe2b3795dd-dshm\") pod \"e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r\" (UID: \"1eae0070-4b15-449f-88df-f8fe2b3795dd\") " pod="llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r" Jul 31 19:08:13.048644 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:13.048540 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eae0070-4b15-449f-88df-f8fe2b3795dd-dshm\") pod \"e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r\" (UID: \"1eae0070-4b15-449f-88df-f8fe2b3795dd\") " pod="llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r" Jul 31 19:08:13.048644 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:13.048633 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eae0070-4b15-449f-88df-f8fe2b3795dd-model-cache\") pod \"e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r\" (UID: \"1eae0070-4b15-449f-88df-f8fe2b3795dd\") " pod="llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r" Jul 31 19:08:13.048873 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:13.048669 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eae0070-4b15-449f-88df-f8fe2b3795dd-tmp-dir\") pod \"e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r\" (UID: \"1eae0070-4b15-449f-88df-f8fe2b3795dd\") " pod="llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r" Jul 31 19:08:13.048873 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:13.048701 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eae0070-4b15-449f-88df-f8fe2b3795dd-home\") pod \"e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r\" (UID: \"1eae0070-4b15-449f-88df-f8fe2b3795dd\") " pod="llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r" Jul 31 19:08:13.048873 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:13.048740 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-877nk\" (UniqueName: \"kubernetes.io/projected/1eae0070-4b15-449f-88df-f8fe2b3795dd-kube-api-access-877nk\") pod \"e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r\" (UID: \"1eae0070-4b15-449f-88df-f8fe2b3795dd\") " pod="llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r" Jul 31 19:08:13.048873 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:13.048782 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eae0070-4b15-449f-88df-f8fe2b3795dd-tls-certs\") pod \"e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r\" (UID: \"1eae0070-4b15-449f-88df-f8fe2b3795dd\") " pod="llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r" Jul 31 19:08:13.049199 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:13.049172 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eae0070-4b15-449f-88df-f8fe2b3795dd-tmp-dir\") pod \"e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r\" (UID: \"1eae0070-4b15-449f-88df-f8fe2b3795dd\") " pod="llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r" Jul 31 19:08:13.049494 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:13.049447 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eae0070-4b15-449f-88df-f8fe2b3795dd-home\") pod \"e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r\" (UID: \"1eae0070-4b15-449f-88df-f8fe2b3795dd\") " pod="llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r" Jul 31 19:08:13.049645 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:13.049622 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eae0070-4b15-449f-88df-f8fe2b3795dd-model-cache\") pod \"e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r\" (UID: \"1eae0070-4b15-449f-88df-f8fe2b3795dd\") " pod="llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r" Jul 31 19:08:13.050961 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:13.050930 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eae0070-4b15-449f-88df-f8fe2b3795dd-dshm\") pod \"e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r\" (UID: \"1eae0070-4b15-449f-88df-f8fe2b3795dd\") " pod="llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r" Jul 31 19:08:13.051774 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:13.051750 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eae0070-4b15-449f-88df-f8fe2b3795dd-tls-certs\") pod \"e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r\" (UID: \"1eae0070-4b15-449f-88df-f8fe2b3795dd\") " pod="llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r" Jul 31 19:08:13.058615 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:13.058589 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-877nk\" (UniqueName: \"kubernetes.io/projected/1eae0070-4b15-449f-88df-f8fe2b3795dd-kube-api-access-877nk\") pod \"e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r\" (UID: \"1eae0070-4b15-449f-88df-f8fe2b3795dd\") " pod="llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r" Jul 31 19:08:13.107613 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:13.107586 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r" Jul 31 19:08:13.254799 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:13.254760 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r"] Jul 31 19:08:13.258126 ip-10-0-139-39 kubenswrapper[2565]: W0731 19:08:13.258096 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1eae0070_4b15_449f_88df_f8fe2b3795dd.slice/crio-8d9d17748fcfd54d83cd567f2230f73258538bd0f3c65f6c956e3abbad735dee WatchSource:0}: Error finding container 8d9d17748fcfd54d83cd567f2230f73258538bd0f3c65f6c956e3abbad735dee: Status 404 returned error can't find the container with id 8d9d17748fcfd54d83cd567f2230f73258538bd0f3c65f6c956e3abbad735dee Jul 31 19:08:13.907918 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:13.907839 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd" Jul 31 19:08:13.907918 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:13.907875 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd" Jul 31 19:08:13.920414 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:13.920385 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd" Jul 31 19:08:14.001823 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:14.001791 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r" event={"ID":"1eae0070-4b15-449f-88df-f8fe2b3795dd","Type":"ContainerStarted","Data":"494557a63e6f56600876413d05a84e365406c3d145a3fbb4fc91978e707c57dd"} Jul 31 19:08:14.001823 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:14.001827 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r" event={"ID":"1eae0070-4b15-449f-88df-f8fe2b3795dd","Type":"ContainerStarted","Data":"8d9d17748fcfd54d83cd567f2230f73258538bd0f3c65f6c956e3abbad735dee"} Jul 31 19:08:14.012953 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:14.012928 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd" Jul 31 19:08:14.022275 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:14.022222 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r" podStartSLOduration=2.022208373 podStartE2EDuration="2.022208373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:08:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:08:14.020551384 +0000 UTC m=+1470.658541183" watchObservedRunningTime="2026-07-31 19:08:14.022208373 +0000 UTC m=+1470.660198173" Jul 31 19:08:20.954158 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:20.954127 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:08:23.107684 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:23.107651 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r" Jul 31 19:08:23.108061 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:23.107703 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r" Jul 31 19:08:23.119758 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:23.119733 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r" Jul 31 19:08:24.068812 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:24.068783 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r" Jul 31 19:08:37.411582 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:37.411550 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd"] Jul 31 19:08:37.438636 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:37.438606 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:08:39.195886 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:39.195851 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r"] Jul 31 19:08:39.220591 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:39.220555 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:08:39.262665 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:39.262623 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:08:50.427264 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:50.427220 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:08:50.863038 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:50.862993 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:08:52.431937 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:52.431879 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd" podUID="7c3b548c-246e-4bdf-9a86-f318a77e1a7f" containerName="main" containerID="cri-o://9b7302dd865848a6fd792ea0e835eb341e875a62485f8b984f56927b4c91b217" gracePeriod=45 Jul 31 19:08:52.684713 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:52.684656 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd" Jul 31 19:08:52.819021 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:52.818985 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnrlq\" (UniqueName: \"kubernetes.io/projected/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-kube-api-access-vnrlq\") pod \"7c3b548c-246e-4bdf-9a86-f318a77e1a7f\" (UID: \"7c3b548c-246e-4bdf-9a86-f318a77e1a7f\") " Jul 31 19:08:52.819187 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:52.819045 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-home\") pod \"7c3b548c-246e-4bdf-9a86-f318a77e1a7f\" (UID: \"7c3b548c-246e-4bdf-9a86-f318a77e1a7f\") " Jul 31 19:08:52.819187 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:52.819072 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-tmp-dir\") pod \"7c3b548c-246e-4bdf-9a86-f318a77e1a7f\" (UID: \"7c3b548c-246e-4bdf-9a86-f318a77e1a7f\") " Jul 31 19:08:52.819187 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:52.819119 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-model-cache\") pod \"7c3b548c-246e-4bdf-9a86-f318a77e1a7f\" (UID: \"7c3b548c-246e-4bdf-9a86-f318a77e1a7f\") " Jul 31 19:08:52.819187 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:52.819164 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-dshm\") pod \"7c3b548c-246e-4bdf-9a86-f318a77e1a7f\" (UID: \"7c3b548c-246e-4bdf-9a86-f318a77e1a7f\") " Jul 31 19:08:52.819444 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:52.819211 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-tls-certs\") pod \"7c3b548c-246e-4bdf-9a86-f318a77e1a7f\" (UID: \"7c3b548c-246e-4bdf-9a86-f318a77e1a7f\") " Jul 31 19:08:52.819444 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:52.819403 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7c3b548c-246e-4bdf-9a86-f318a77e1a7f" (UID: "7c3b548c-246e-4bdf-9a86-f318a77e1a7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:08:52.819444 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:52.819413 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-home" (OuterVolumeSpecName: "home") pod "7c3b548c-246e-4bdf-9a86-f318a77e1a7f" (UID: "7c3b548c-246e-4bdf-9a86-f318a77e1a7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:08:52.819444 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:52.819429 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-model-cache" (OuterVolumeSpecName: "model-cache") pod "7c3b548c-246e-4bdf-9a86-f318a77e1a7f" (UID: "7c3b548c-246e-4bdf-9a86-f318a77e1a7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:08:52.819645 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:52.819597 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:08:52.819645 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:52.819618 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:08:52.819645 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:52.819632 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:08:52.821357 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:52.821329 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c3b548c-246e-4bdf-9a86-f318a77e1a7f" (UID: "7c3b548c-246e-4bdf-9a86-f318a77e1a7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 19:08:52.821484 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:52.821369 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-dshm" (OuterVolumeSpecName: "dshm") pod "7c3b548c-246e-4bdf-9a86-f318a77e1a7f" (UID: "7c3b548c-246e-4bdf-9a86-f318a77e1a7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:08:52.821484 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:52.821473 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-kube-api-access-vnrlq" (OuterVolumeSpecName: "kube-api-access-vnrlq") pod "7c3b548c-246e-4bdf-9a86-f318a77e1a7f" (UID: "7c3b548c-246e-4bdf-9a86-f318a77e1a7f"). InnerVolumeSpecName "kube-api-access-vnrlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:08:52.920710 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:52.920668 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:08:52.920710 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:52.920710 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:08:52.920710 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:52.920727 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnrlq\" (UniqueName: \"kubernetes.io/projected/7c3b548c-246e-4bdf-9a86-f318a77e1a7f-kube-api-access-vnrlq\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:08:53.204666 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:53.204630 2565 generic.go:356] "Generic (PLEG): container finished" podID="7c3b548c-246e-4bdf-9a86-f318a77e1a7f" containerID="9b7302dd865848a6fd792ea0e835eb341e875a62485f8b984f56927b4c91b217" exitCode=0 Jul 31 19:08:53.204838 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:53.204676 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd" Jul 31 19:08:53.204838 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:53.204675 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd" event={"ID":"7c3b548c-246e-4bdf-9a86-f318a77e1a7f","Type":"ContainerDied","Data":"9b7302dd865848a6fd792ea0e835eb341e875a62485f8b984f56927b4c91b217"} Jul 31 19:08:53.204838 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:53.204775 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd" event={"ID":"7c3b548c-246e-4bdf-9a86-f318a77e1a7f","Type":"ContainerDied","Data":"bc6be7839e74f1c777c6ab866fdb22967ec895e31e3bb96fdeeaedaa83a2426c"} Jul 31 19:08:53.204838 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:53.204795 2565 scope.go:117] "RemoveContainer" containerID="9b7302dd865848a6fd792ea0e835eb341e875a62485f8b984f56927b4c91b217" Jul 31 19:08:53.213706 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:53.213687 2565 scope.go:117] "RemoveContainer" containerID="9b7302dd865848a6fd792ea0e835eb341e875a62485f8b984f56927b4c91b217" Jul 31 19:08:53.213953 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:08:53.213928 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b7302dd865848a6fd792ea0e835eb341e875a62485f8b984f56927b4c91b217\": container with ID starting with 9b7302dd865848a6fd792ea0e835eb341e875a62485f8b984f56927b4c91b217 not found: ID does not exist" containerID="9b7302dd865848a6fd792ea0e835eb341e875a62485f8b984f56927b4c91b217" Jul 31 19:08:53.214001 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:53.213960 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b7302dd865848a6fd792ea0e835eb341e875a62485f8b984f56927b4c91b217"} err="failed to get container status \"9b7302dd865848a6fd792ea0e835eb341e875a62485f8b984f56927b4c91b217\": rpc error: code = NotFound desc = could not find container \"9b7302dd865848a6fd792ea0e835eb341e875a62485f8b984f56927b4c91b217\": container with ID starting with 9b7302dd865848a6fd792ea0e835eb341e875a62485f8b984f56927b4c91b217 not found: ID does not exist" Jul 31 19:08:53.227097 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:53.227073 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd"] Jul 31 19:08:53.233364 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:53.233343 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-f814dccf-kserve-85b8b5f7d7-cbnwd"] Jul 31 19:08:53.891996 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:53.891960 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c3b548c-246e-4bdf-9a86-f318a77e1a7f" path="/var/lib/kubelet/pods/7c3b548c-246e-4bdf-9a86-f318a77e1a7f/volumes" Jul 31 19:08:54.214546 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:54.214459 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r" podUID="1eae0070-4b15-449f-88df-f8fe2b3795dd" containerName="main" containerID="cri-o://494557a63e6f56600876413d05a84e365406c3d145a3fbb4fc91978e707c57dd" gracePeriod=45 Jul 31 19:08:54.466987 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:54.466929 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r" Jul 31 19:08:54.636325 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:54.636294 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eae0070-4b15-449f-88df-f8fe2b3795dd-tls-certs\") pod \"1eae0070-4b15-449f-88df-f8fe2b3795dd\" (UID: \"1eae0070-4b15-449f-88df-f8fe2b3795dd\") " Jul 31 19:08:54.636490 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:54.636342 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eae0070-4b15-449f-88df-f8fe2b3795dd-model-cache\") pod \"1eae0070-4b15-449f-88df-f8fe2b3795dd\" (UID: \"1eae0070-4b15-449f-88df-f8fe2b3795dd\") " Jul 31 19:08:54.636490 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:54.636393 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eae0070-4b15-449f-88df-f8fe2b3795dd-dshm\") pod \"1eae0070-4b15-449f-88df-f8fe2b3795dd\" (UID: \"1eae0070-4b15-449f-88df-f8fe2b3795dd\") " Jul 31 19:08:54.636490 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:54.636445 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eae0070-4b15-449f-88df-f8fe2b3795dd-home\") pod \"1eae0070-4b15-449f-88df-f8fe2b3795dd\" (UID: \"1eae0070-4b15-449f-88df-f8fe2b3795dd\") " Jul 31 19:08:54.636490 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:54.636486 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eae0070-4b15-449f-88df-f8fe2b3795dd-tmp-dir\") pod \"1eae0070-4b15-449f-88df-f8fe2b3795dd\" (UID: \"1eae0070-4b15-449f-88df-f8fe2b3795dd\") " Jul 31 19:08:54.636714 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:54.636533 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-877nk\" (UniqueName: \"kubernetes.io/projected/1eae0070-4b15-449f-88df-f8fe2b3795dd-kube-api-access-877nk\") pod \"1eae0070-4b15-449f-88df-f8fe2b3795dd\" (UID: \"1eae0070-4b15-449f-88df-f8fe2b3795dd\") " Jul 31 19:08:54.636714 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:54.636689 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eae0070-4b15-449f-88df-f8fe2b3795dd-model-cache" (OuterVolumeSpecName: "model-cache") pod "1eae0070-4b15-449f-88df-f8fe2b3795dd" (UID: "1eae0070-4b15-449f-88df-f8fe2b3795dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:08:54.636817 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:54.636782 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eae0070-4b15-449f-88df-f8fe2b3795dd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1eae0070-4b15-449f-88df-f8fe2b3795dd" (UID: "1eae0070-4b15-449f-88df-f8fe2b3795dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:08:54.636817 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:54.636794 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eae0070-4b15-449f-88df-f8fe2b3795dd-home" (OuterVolumeSpecName: "home") pod "1eae0070-4b15-449f-88df-f8fe2b3795dd" (UID: "1eae0070-4b15-449f-88df-f8fe2b3795dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:08:54.636910 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:54.636894 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eae0070-4b15-449f-88df-f8fe2b3795dd-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:08:54.636968 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:54.636935 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eae0070-4b15-449f-88df-f8fe2b3795dd-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:08:54.636968 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:54.636946 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eae0070-4b15-449f-88df-f8fe2b3795dd-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:08:54.638830 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:54.638805 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eae0070-4b15-449f-88df-f8fe2b3795dd-kube-api-access-877nk" (OuterVolumeSpecName: "kube-api-access-877nk") pod "1eae0070-4b15-449f-88df-f8fe2b3795dd" (UID: "1eae0070-4b15-449f-88df-f8fe2b3795dd"). InnerVolumeSpecName "kube-api-access-877nk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:08:54.638977 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:54.638961 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eae0070-4b15-449f-88df-f8fe2b3795dd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1eae0070-4b15-449f-88df-f8fe2b3795dd" (UID: "1eae0070-4b15-449f-88df-f8fe2b3795dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 19:08:54.638977 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:54.638966 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eae0070-4b15-449f-88df-f8fe2b3795dd-dshm" (OuterVolumeSpecName: "dshm") pod "1eae0070-4b15-449f-88df-f8fe2b3795dd" (UID: "1eae0070-4b15-449f-88df-f8fe2b3795dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:08:54.737786 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:54.737696 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eae0070-4b15-449f-88df-f8fe2b3795dd-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:08:54.737786 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:54.737730 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-877nk\" (UniqueName: \"kubernetes.io/projected/1eae0070-4b15-449f-88df-f8fe2b3795dd-kube-api-access-877nk\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:08:54.737786 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:54.737741 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eae0070-4b15-449f-88df-f8fe2b3795dd-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:08:54.857504 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:54.857465 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:08:55.216080 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:55.216041 2565 generic.go:356] "Generic (PLEG): container finished" podID="1eae0070-4b15-449f-88df-f8fe2b3795dd" containerID="494557a63e6f56600876413d05a84e365406c3d145a3fbb4fc91978e707c57dd" exitCode=0 Jul 31 19:08:55.216524 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:55.216087 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r" Jul 31 19:08:55.216524 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:55.216127 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r" event={"ID":"1eae0070-4b15-449f-88df-f8fe2b3795dd","Type":"ContainerDied","Data":"494557a63e6f56600876413d05a84e365406c3d145a3fbb4fc91978e707c57dd"} Jul 31 19:08:55.216524 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:55.216176 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r" event={"ID":"1eae0070-4b15-449f-88df-f8fe2b3795dd","Type":"ContainerDied","Data":"8d9d17748fcfd54d83cd567f2230f73258538bd0f3c65f6c956e3abbad735dee"} Jul 31 19:08:55.216524 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:55.216194 2565 scope.go:117] "RemoveContainer" containerID="494557a63e6f56600876413d05a84e365406c3d145a3fbb4fc91978e707c57dd" Jul 31 19:08:55.225753 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:55.225735 2565 scope.go:117] "RemoveContainer" containerID="494557a63e6f56600876413d05a84e365406c3d145a3fbb4fc91978e707c57dd" Jul 31 19:08:55.225994 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:08:55.225974 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"494557a63e6f56600876413d05a84e365406c3d145a3fbb4fc91978e707c57dd\": container with ID starting with 494557a63e6f56600876413d05a84e365406c3d145a3fbb4fc91978e707c57dd not found: ID does not exist" containerID="494557a63e6f56600876413d05a84e365406c3d145a3fbb4fc91978e707c57dd" Jul 31 19:08:55.226061 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:55.226002 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"494557a63e6f56600876413d05a84e365406c3d145a3fbb4fc91978e707c57dd"} err="failed to get container status \"494557a63e6f56600876413d05a84e365406c3d145a3fbb4fc91978e707c57dd\": rpc error: code = NotFound desc = could not find container \"494557a63e6f56600876413d05a84e365406c3d145a3fbb4fc91978e707c57dd\": container with ID starting with 494557a63e6f56600876413d05a84e365406c3d145a3fbb4fc91978e707c57dd not found: ID does not exist" Jul 31 19:08:55.240143 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:55.240120 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r"] Jul 31 19:08:55.242624 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:55.242601 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-f814dccf-kserve-66cf8bc459-drk8r"] Jul 31 19:08:55.889570 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:08:55.889534 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1eae0070-4b15-449f-88df-f8fe2b3795dd" path="/var/lib/kubelet/pods/1eae0070-4b15-449f-88df-f8fe2b3795dd/volumes" Jul 31 19:09:01.059558 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:09:01.059512 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:09:11.455636 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:09:11.455556 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:09:17.757966 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:09:17.757924 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:09:24.366921 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:09:24.366890 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:09:30.654088 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:09:30.654044 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:09:35.959456 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:09:35.959424 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:09:46.155106 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:09:46.155067 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:09:52.555150 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:09:52.555109 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:10:09.060108 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:10:09.060073 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:10:15.851362 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:10:15.851329 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:10:20.253653 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:10:20.253619 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:10:21.188167 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:10:21.188133 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:10:27.284748 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:10:27.284715 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:10:34.851619 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:10:34.851583 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:10:41.155048 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:10:41.154959 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:10:46.148478 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:10:46.148442 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:10:47.133985 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:10:47.133948 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:10:52.561677 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:10:52.561633 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:11:02.369516 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:11:02.369481 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:11:08.965710 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:11:08.965668 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:11:19.252429 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:11:19.252391 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:11:25.962607 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:11:25.962566 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:11:36.353121 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:11:36.353080 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:11:51.658957 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:11:51.658918 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:11:57.555506 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:11:57.555471 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:12:01.059441 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:01.059405 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:12:03.910856 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:03.910824 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:12:15.360819 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:15.360738 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:12:23.865796 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:23.865763 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:12:26.995795 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:26.995761 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:12:29.452934 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:29.452894 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:12:30.042796 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:30.042759 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:12:30.134586 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:30.134553 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:12:30.537634 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:30.537601 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:12:36.586738 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:36.586703 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:12:36.687300 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:36.687264 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:12:36.824827 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:36.824790 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:12:37.112604 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.112572 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l"] Jul 31 19:12:37.112994 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.112981 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c3b548c-246e-4bdf-9a86-f318a77e1a7f" containerName="main" Jul 31 19:12:37.113039 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.112996 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c3b548c-246e-4bdf-9a86-f318a77e1a7f" containerName="main" Jul 31 19:12:37.113039 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.113007 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eae0070-4b15-449f-88df-f8fe2b3795dd" containerName="main" Jul 31 19:12:37.113039 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.113013 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eae0070-4b15-449f-88df-f8fe2b3795dd" containerName="main" Jul 31 19:12:37.113140 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.113071 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c3b548c-246e-4bdf-9a86-f318a77e1a7f" containerName="main" Jul 31 19:12:37.113140 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.113081 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1eae0070-4b15-449f-88df-f8fe2b3795dd" containerName="main" Jul 31 19:12:37.117493 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.117472 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:37.119882 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.119862 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b30c0866-openshift-default-dockercfg-snxnv\"" Jul 31 19:12:37.142385 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.142347 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l"] Jul 31 19:12:37.144286 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.144228 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:12:37.270462 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.270427 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfqzl\" (UniqueName: \"kubernetes.io/projected/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-kube-api-access-xfqzl\") pod \"e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:37.270652 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.270478 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-istio-data\") pod \"e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:37.270652 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.270518 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-istio-podinfo\") pod \"e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:37.270652 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.270534 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-istio-token\") pod \"e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:37.270652 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.270609 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-workload-certs\") pod \"e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:37.270842 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.270666 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-credential-socket\") pod \"e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:37.270842 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.270683 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-istio-envoy\") pod \"e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:37.270842 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.270709 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-istiod-ca-cert\") pod \"e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:37.270842 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.270759 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-workload-socket\") pod \"e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:37.282688 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.282658 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:12:37.371746 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.371657 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfqzl\" (UniqueName: \"kubernetes.io/projected/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-kube-api-access-xfqzl\") pod \"e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:37.371746 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.371697 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-istio-data\") pod \"e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:37.371746 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.371722 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-istio-podinfo\") pod \"e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:37.371746 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.371742 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-istio-token\") pod \"e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:37.372092 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.371784 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-workload-certs\") pod \"e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:37.372092 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.371834 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-credential-socket\") pod \"e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:37.372092 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.371856 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-istio-envoy\") pod \"e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:37.372092 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.371884 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-istiod-ca-cert\") pod \"e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:37.372092 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.371942 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-workload-socket\") pod \"e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:37.372396 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.372199 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-workload-certs\") pod \"e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:37.372396 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.372385 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-workload-socket\") pod \"e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:37.372507 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.372454 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-istio-data\") pod \"e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:37.372551 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.372531 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-credential-socket\") pod \"e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:37.372663 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.372647 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-istiod-ca-cert\") pod \"e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:37.374287 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.374267 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-istio-envoy\") pod \"e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:37.374389 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.374369 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-istio-podinfo\") pod \"e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:37.382066 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.382044 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-istio-token\") pod \"e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:37.382527 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.382505 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfqzl\" (UniqueName: \"kubernetes.io/projected/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-kube-api-access-xfqzl\") pod \"e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:37.427988 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.427969 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:37.567539 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.567502 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l"] Jul 31 19:12:37.569731 ip-10-0-139-39 kubenswrapper[2565]: W0731 19:12:37.569708 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd82a326c_fa0e_4775_bdb6_d67c7513c1d7.slice/crio-78b4a2d0cefba02cb3258aaec079b1450a994c117a18dddc27d4634d4c54a3df WatchSource:0}: Error finding container 78b4a2d0cefba02cb3258aaec079b1450a994c117a18dddc27d4634d4c54a3df: Status 404 returned error can't find the container with id 78b4a2d0cefba02cb3258aaec079b1450a994c117a18dddc27d4634d4c54a3df Jul 31 19:12:37.571920 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.571886 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:12:37.571990 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.571948 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:12:37.571990 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:37.571977 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:12:38.329028 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:38.328989 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" event={"ID":"d82a326c-fa0e-4775-bdb6-d67c7513c1d7","Type":"ContainerStarted","Data":"c5134dc60192a9de54f51cb5a50b952a32de5e7ea97589601ce90caecedc680f"} Jul 31 19:12:38.329449 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:38.329045 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" event={"ID":"d82a326c-fa0e-4775-bdb6-d67c7513c1d7","Type":"ContainerStarted","Data":"78b4a2d0cefba02cb3258aaec079b1450a994c117a18dddc27d4634d4c54a3df"} Jul 31 19:12:38.352587 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:38.352537 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" podStartSLOduration=1.35252091 podStartE2EDuration="1.35252091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:12:38.348752518 +0000 UTC m=+1734.986742318" watchObservedRunningTime="2026-07-31 19:12:38.35252091 +0000 UTC m=+1734.990510710" Jul 31 19:12:38.428331 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:38.428295 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:38.433498 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:38.433475 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:39.333529 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:39.333497 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:39.334433 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:39.334413 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:39.729622 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:39.729527 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:12:39.826789 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:39.826753 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:12:40.877460 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:40.877430 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:12:41.037665 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:41.037627 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:12:41.399827 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:41.399794 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:12:41.495438 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:41.495397 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:12:42.855309 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:42.855260 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:12:53.404543 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:53.404506 2565 generic.go:356] "Generic (PLEG): container finished" podID="d82a326c-fa0e-4775-bdb6-d67c7513c1d7" containerID="c5134dc60192a9de54f51cb5a50b952a32de5e7ea97589601ce90caecedc680f" exitCode=0 Jul 31 19:12:53.405090 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:53.404583 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" event={"ID":"d82a326c-fa0e-4775-bdb6-d67c7513c1d7","Type":"ContainerDied","Data":"c5134dc60192a9de54f51cb5a50b952a32de5e7ea97589601ce90caecedc680f"} Jul 31 19:12:53.405167 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:53.405105 2565 scope.go:117] "RemoveContainer" containerID="c5134dc60192a9de54f51cb5a50b952a32de5e7ea97589601ce90caecedc680f" Jul 31 19:12:53.406185 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:53.406158 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:12:53.406266 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:53.406227 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:12:53.406311 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:53.406270 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:12:54.343261 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:54.340869 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:54.411768 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:54.411732 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" event={"ID":"d82a326c-fa0e-4775-bdb6-d67c7513c1d7","Type":"ContainerStarted","Data":"54c282fa39d141ae04e0eeb58e22d7e4b7b23d16a8ad164cf2718904b69f32ea"} Jul 31 19:12:54.428893 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:54.428860 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:54.430537 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:54.430508 2565 patch_prober.go:28] interesting pod/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 31 19:12:54.430679 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:54.430568 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" podUID="d82a326c-fa0e-4775-bdb6-d67c7513c1d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 31 19:12:55.431621 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:55.431584 2565 patch_prober.go:28] interesting pod/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 31 19:12:55.432683 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:55.432288 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" podUID="d82a326c-fa0e-4775-bdb6-d67c7513c1d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 31 19:12:56.429370 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:56.429327 2565 patch_prober.go:28] interesting pod/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 31 19:12:56.429581 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:56.429390 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" podUID="d82a326c-fa0e-4775-bdb6-d67c7513c1d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 31 19:12:57.429184 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:57.429136 2565 patch_prober.go:28] interesting pod/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 31 19:12:57.429700 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:57.429203 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" podUID="d82a326c-fa0e-4775-bdb6-d67c7513c1d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 31 19:12:58.429450 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:58.429412 2565 patch_prober.go:28] interesting pod/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 31 19:12:58.429938 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:58.429480 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" podUID="d82a326c-fa0e-4775-bdb6-d67c7513c1d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 31 19:12:59.432710 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:59.432678 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:59.433084 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:59.432948 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:12:59.433717 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:12:59.433696 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:13:21.529257 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:21.529210 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:13:21.890853 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:21.890760 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l"] Jul 31 19:13:21.891024 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:21.890971 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" podUID="d82a326c-fa0e-4775-bdb6-d67c7513c1d7" containerName="istio-proxy" containerID="cri-o://54c282fa39d141ae04e0eeb58e22d7e4b7b23d16a8ad164cf2718904b69f32ea" gracePeriod=30 Jul 31 19:13:22.872549 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:22.872513 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr"] Jul 31 19:13:22.875233 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:22.875212 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:22.877334 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:22.877311 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-b9249e59-gw-openshift-default-dockercfg-wdq5f\"" Jul 31 19:13:22.889260 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:22.889222 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr"] Jul 31 19:13:23.023444 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.023412 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e147296-228d-4e0c-a8c3-5998f22f46f9-istiod-ca-cert\") pod \"e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:23.023601 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.023471 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e147296-228d-4e0c-a8c3-5998f22f46f9-credential-socket\") pod \"e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:23.023601 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.023521 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e147296-228d-4e0c-a8c3-5998f22f46f9-istio-podinfo\") pod \"e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:23.023601 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.023546 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e147296-228d-4e0c-a8c3-5998f22f46f9-istio-data\") pod \"e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:23.023601 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.023572 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e147296-228d-4e0c-a8c3-5998f22f46f9-istio-token\") pod \"e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:23.023601 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.023593 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e147296-228d-4e0c-a8c3-5998f22f46f9-workload-certs\") pod \"e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:23.023794 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.023626 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e147296-228d-4e0c-a8c3-5998f22f46f9-workload-socket\") pod \"e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:23.023794 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.023678 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e147296-228d-4e0c-a8c3-5998f22f46f9-istio-envoy\") pod \"e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:23.023794 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.023721 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znzt4\" (UniqueName: \"kubernetes.io/projected/7e147296-228d-4e0c-a8c3-5998f22f46f9-kube-api-access-znzt4\") pod \"e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:23.125098 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.125000 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e147296-228d-4e0c-a8c3-5998f22f46f9-istio-podinfo\") pod \"e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:23.125098 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.125056 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e147296-228d-4e0c-a8c3-5998f22f46f9-istio-data\") pod \"e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:23.125098 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.125095 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e147296-228d-4e0c-a8c3-5998f22f46f9-istio-token\") pod \"e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:23.125421 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.125130 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e147296-228d-4e0c-a8c3-5998f22f46f9-workload-certs\") pod \"e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:23.125421 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.125282 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e147296-228d-4e0c-a8c3-5998f22f46f9-workload-socket\") pod \"e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:23.125421 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.125330 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e147296-228d-4e0c-a8c3-5998f22f46f9-istio-envoy\") pod \"e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:23.125421 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.125349 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znzt4\" (UniqueName: \"kubernetes.io/projected/7e147296-228d-4e0c-a8c3-5998f22f46f9-kube-api-access-znzt4\") pod \"e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:23.125421 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.125383 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e147296-228d-4e0c-a8c3-5998f22f46f9-istiod-ca-cert\") pod \"e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:23.125766 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.125457 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e147296-228d-4e0c-a8c3-5998f22f46f9-credential-socket\") pod \"e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:23.125766 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.125553 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e147296-228d-4e0c-a8c3-5998f22f46f9-workload-certs\") pod \"e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:23.125766 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.125620 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e147296-228d-4e0c-a8c3-5998f22f46f9-istio-data\") pod \"e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:23.125766 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.125734 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e147296-228d-4e0c-a8c3-5998f22f46f9-workload-socket\") pod \"e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:23.125971 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.125782 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e147296-228d-4e0c-a8c3-5998f22f46f9-credential-socket\") pod \"e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:23.126158 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.126139 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e147296-228d-4e0c-a8c3-5998f22f46f9-istiod-ca-cert\") pod \"e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:23.127676 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.127643 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e147296-228d-4e0c-a8c3-5998f22f46f9-istio-podinfo\") pod \"e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:23.127795 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.127786 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e147296-228d-4e0c-a8c3-5998f22f46f9-istio-envoy\") pod \"e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:23.136713 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.136684 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:13:23.137221 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.137194 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e147296-228d-4e0c-a8c3-5998f22f46f9-istio-token\") pod \"e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:23.137550 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.137533 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znzt4\" (UniqueName: \"kubernetes.io/projected/7e147296-228d-4e0c-a8c3-5998f22f46f9-kube-api-access-znzt4\") pod \"e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:23.185437 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.185406 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:23.331533 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.331507 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr"] Jul 31 19:13:23.333991 ip-10-0-139-39 kubenswrapper[2565]: W0731 19:13:23.333970 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e147296_228d_4e0c_a8c3_5998f22f46f9.slice/crio-cedf80c555f4d7f42ad004e9758a95e3b2a279c9fdd853558707e80b2479d599 WatchSource:0}: Error finding container cedf80c555f4d7f42ad004e9758a95e3b2a279c9fdd853558707e80b2479d599: Status 404 returned error can't find the container with id cedf80c555f4d7f42ad004e9758a95e3b2a279c9fdd853558707e80b2479d599 Jul 31 19:13:23.336077 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.336044 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:13:23.336166 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.336109 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:13:23.336166 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.336136 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:13:23.561945 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.561898 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" event={"ID":"7e147296-228d-4e0c-a8c3-5998f22f46f9","Type":"ContainerStarted","Data":"23324c75840ed6a649e097db0a30e5a06cf08fc605d3eec04a62424cf077703f"} Jul 31 19:13:23.562106 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.561964 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" event={"ID":"7e147296-228d-4e0c-a8c3-5998f22f46f9","Type":"ContainerStarted","Data":"cedf80c555f4d7f42ad004e9758a95e3b2a279c9fdd853558707e80b2479d599"} Jul 31 19:13:23.582093 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:23.582039 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" podStartSLOduration=1.582018128 podStartE2EDuration="1.582018128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:13:23.580137918 +0000 UTC m=+1780.218127729" watchObservedRunningTime="2026-07-31 19:13:23.582018128 +0000 UTC m=+1780.220007923" Jul 31 19:13:24.185597 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:24.185563 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:24.190337 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:24.190313 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:24.566931 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:24.566890 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:24.567715 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:24.567693 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:24.750670 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:24.750637 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:13:25.054679 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:25.054647 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:13:25.163678 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:25.163648 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:13:25.262813 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:25.262767 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:13:25.859806 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:25.859772 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:13:25.931097 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:25.931026 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:13:27.144817 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.144791 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:13:27.270274 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.270155 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-istio-token\") pod \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " Jul 31 19:13:27.270274 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.270216 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-istio-envoy\") pod \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " Jul 31 19:13:27.270524 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.270294 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-workload-socket\") pod \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " Jul 31 19:13:27.270524 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.270355 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-istio-data\") pod \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " Jul 31 19:13:27.270524 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.270384 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-workload-certs\") pod \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " Jul 31 19:13:27.270524 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.270419 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-istio-podinfo\") pod \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " Jul 31 19:13:27.270524 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.270462 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-credential-socket\") pod \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " Jul 31 19:13:27.270524 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.270503 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfqzl\" (UniqueName: \"kubernetes.io/projected/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-kube-api-access-xfqzl\") pod \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " Jul 31 19:13:27.270834 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.270534 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-istiod-ca-cert\") pod \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\" (UID: \"d82a326c-fa0e-4775-bdb6-d67c7513c1d7\") " Jul 31 19:13:27.270834 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.270670 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-istio-data" (OuterVolumeSpecName: "istio-data") pod "d82a326c-fa0e-4775-bdb6-d67c7513c1d7" (UID: "d82a326c-fa0e-4775-bdb6-d67c7513c1d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:13:27.270834 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.270708 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d82a326c-fa0e-4775-bdb6-d67c7513c1d7" (UID: "d82a326c-fa0e-4775-bdb6-d67c7513c1d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:13:27.270834 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.270763 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d82a326c-fa0e-4775-bdb6-d67c7513c1d7" (UID: "d82a326c-fa0e-4775-bdb6-d67c7513c1d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:13:27.271042 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.270923 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-workload-socket\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:13:27.271042 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.270944 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-istio-data\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:13:27.271042 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.270958 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-credential-socket\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:13:27.271042 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.271006 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d82a326c-fa0e-4775-bdb6-d67c7513c1d7" (UID: "d82a326c-fa0e-4775-bdb6-d67c7513c1d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 19:13:27.271266 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.271041 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d82a326c-fa0e-4775-bdb6-d67c7513c1d7" (UID: "d82a326c-fa0e-4775-bdb6-d67c7513c1d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:13:27.272577 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.272544 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d82a326c-fa0e-4775-bdb6-d67c7513c1d7" (UID: "d82a326c-fa0e-4775-bdb6-d67c7513c1d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:13:27.272577 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.272551 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d82a326c-fa0e-4775-bdb6-d67c7513c1d7" (UID: "d82a326c-fa0e-4775-bdb6-d67c7513c1d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 19:13:27.272764 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.272649 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-istio-token" (OuterVolumeSpecName: "istio-token") pod "d82a326c-fa0e-4775-bdb6-d67c7513c1d7" (UID: "d82a326c-fa0e-4775-bdb6-d67c7513c1d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:13:27.272764 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.272674 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-kube-api-access-xfqzl" (OuterVolumeSpecName: "kube-api-access-xfqzl") pod "d82a326c-fa0e-4775-bdb6-d67c7513c1d7" (UID: "d82a326c-fa0e-4775-bdb6-d67c7513c1d7"). InnerVolumeSpecName "kube-api-access-xfqzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:13:27.372402 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.372370 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-istio-token\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:13:27.372402 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.372396 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-istio-envoy\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:13:27.372402 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.372407 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-workload-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:13:27.372634 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.372415 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-istio-podinfo\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:13:27.372634 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.372424 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfqzl\" (UniqueName: \"kubernetes.io/projected/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-kube-api-access-xfqzl\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:13:27.372634 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.372435 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d82a326c-fa0e-4775-bdb6-d67c7513c1d7-istiod-ca-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:13:27.582272 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.582165 2565 generic.go:356] "Generic (PLEG): container finished" podID="d82a326c-fa0e-4775-bdb6-d67c7513c1d7" containerID="54c282fa39d141ae04e0eeb58e22d7e4b7b23d16a8ad164cf2718904b69f32ea" exitCode=0 Jul 31 19:13:27.582272 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.582211 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" Jul 31 19:13:27.582486 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.582272 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" event={"ID":"d82a326c-fa0e-4775-bdb6-d67c7513c1d7","Type":"ContainerDied","Data":"54c282fa39d141ae04e0eeb58e22d7e4b7b23d16a8ad164cf2718904b69f32ea"} Jul 31 19:13:27.582486 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.582329 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l" event={"ID":"d82a326c-fa0e-4775-bdb6-d67c7513c1d7","Type":"ContainerDied","Data":"78b4a2d0cefba02cb3258aaec079b1450a994c117a18dddc27d4634d4c54a3df"} Jul 31 19:13:27.582486 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.582352 2565 scope.go:117] "RemoveContainer" containerID="54c282fa39d141ae04e0eeb58e22d7e4b7b23d16a8ad164cf2718904b69f32ea" Jul 31 19:13:27.591700 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.591685 2565 scope.go:117] "RemoveContainer" containerID="c5134dc60192a9de54f51cb5a50b952a32de5e7ea97589601ce90caecedc680f" Jul 31 19:13:27.599747 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.599706 2565 scope.go:117] "RemoveContainer" containerID="54c282fa39d141ae04e0eeb58e22d7e4b7b23d16a8ad164cf2718904b69f32ea" Jul 31 19:13:27.599961 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:13:27.599941 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54c282fa39d141ae04e0eeb58e22d7e4b7b23d16a8ad164cf2718904b69f32ea\": container with ID starting with 54c282fa39d141ae04e0eeb58e22d7e4b7b23d16a8ad164cf2718904b69f32ea not found: ID does not exist" containerID="54c282fa39d141ae04e0eeb58e22d7e4b7b23d16a8ad164cf2718904b69f32ea" Jul 31 19:13:27.600004 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.599969 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54c282fa39d141ae04e0eeb58e22d7e4b7b23d16a8ad164cf2718904b69f32ea"} err="failed to get container status \"54c282fa39d141ae04e0eeb58e22d7e4b7b23d16a8ad164cf2718904b69f32ea\": rpc error: code = NotFound desc = could not find container \"54c282fa39d141ae04e0eeb58e22d7e4b7b23d16a8ad164cf2718904b69f32ea\": container with ID starting with 54c282fa39d141ae04e0eeb58e22d7e4b7b23d16a8ad164cf2718904b69f32ea not found: ID does not exist" Jul 31 19:13:27.600004 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.599985 2565 scope.go:117] "RemoveContainer" containerID="c5134dc60192a9de54f51cb5a50b952a32de5e7ea97589601ce90caecedc680f" Jul 31 19:13:27.600210 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:13:27.600193 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5134dc60192a9de54f51cb5a50b952a32de5e7ea97589601ce90caecedc680f\": container with ID starting with c5134dc60192a9de54f51cb5a50b952a32de5e7ea97589601ce90caecedc680f not found: ID does not exist" containerID="c5134dc60192a9de54f51cb5a50b952a32de5e7ea97589601ce90caecedc680f" Jul 31 19:13:27.600275 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.600215 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5134dc60192a9de54f51cb5a50b952a32de5e7ea97589601ce90caecedc680f"} err="failed to get container status \"c5134dc60192a9de54f51cb5a50b952a32de5e7ea97589601ce90caecedc680f\": rpc error: code = NotFound desc = could not find container \"c5134dc60192a9de54f51cb5a50b952a32de5e7ea97589601ce90caecedc680f\": container with ID starting with c5134dc60192a9de54f51cb5a50b952a32de5e7ea97589601ce90caecedc680f not found: ID does not exist" Jul 31 19:13:27.605360 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.605338 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l"] Jul 31 19:13:27.609952 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.609930 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b30c0866-openshift-default-6f9dff7cbd-wwf5l"] Jul 31 19:13:27.886613 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:27.886540 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d82a326c-fa0e-4775-bdb6-d67c7513c1d7" path="/var/lib/kubelet/pods/d82a326c-fa0e-4775-bdb6-d67c7513c1d7/volumes" Jul 31 19:13:32.249681 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:32.249635 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:13:38.646298 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:38.646164 2565 generic.go:356] "Generic (PLEG): container finished" podID="7e147296-228d-4e0c-a8c3-5998f22f46f9" containerID="23324c75840ed6a649e097db0a30e5a06cf08fc605d3eec04a62424cf077703f" exitCode=0 Jul 31 19:13:38.646298 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:38.646255 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" event={"ID":"7e147296-228d-4e0c-a8c3-5998f22f46f9","Type":"ContainerDied","Data":"23324c75840ed6a649e097db0a30e5a06cf08fc605d3eec04a62424cf077703f"} Jul 31 19:13:38.646858 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:38.646803 2565 scope.go:117] "RemoveContainer" containerID="23324c75840ed6a649e097db0a30e5a06cf08fc605d3eec04a62424cf077703f" Jul 31 19:13:38.647871 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:38.647843 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:13:38.647976 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:38.647912 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:13:38.647976 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:38.647943 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:13:39.567252 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:39.567202 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:39.655387 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:39.655353 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" event={"ID":"7e147296-228d-4e0c-a8c3-5998f22f46f9","Type":"ContainerStarted","Data":"261687edf61e8196ee61950df3cd4c32b7c3751fd9f0638005f58ebf1c569065"} Jul 31 19:13:40.186207 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:40.186170 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:40.187091 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:40.187069 2565 patch_prober.go:28] interesting pod/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 31 19:13:40.187144 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:40.187119 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" podUID="7e147296-228d-4e0c-a8c3-5998f22f46f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 31 19:13:41.186308 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:41.186270 2565 patch_prober.go:28] interesting pod/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 31 19:13:41.186702 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:41.186327 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" podUID="7e147296-228d-4e0c-a8c3-5998f22f46f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 31 19:13:42.185915 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:42.185879 2565 patch_prober.go:28] interesting pod/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 31 19:13:42.186067 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:42.185943 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" podUID="7e147296-228d-4e0c-a8c3-5998f22f46f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 31 19:13:43.186664 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:43.186624 2565 patch_prober.go:28] interesting pod/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 31 19:13:43.187041 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:43.186695 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" podUID="7e147296-228d-4e0c-a8c3-5998f22f46f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 31 19:13:44.185862 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:44.185826 2565 patch_prober.go:28] interesting pod/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 31 19:13:44.186031 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:44.185906 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" podUID="7e147296-228d-4e0c-a8c3-5998f22f46f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 31 19:13:45.190298 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:45.190271 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:45.190647 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:45.190515 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:45.191101 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:45.191083 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:13:56.710224 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:56.710181 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:13:56.815075 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:56.815040 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:13:56.894194 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:56.894164 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:13:57.032269 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.031034 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr"] Jul 31 19:13:57.032269 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.031344 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" podUID="7e147296-228d-4e0c-a8c3-5998f22f46f9" containerName="istio-proxy" containerID="cri-o://261687edf61e8196ee61950df3cd4c32b7c3751fd9f0638005f58ebf1c569065" gracePeriod=30 Jul 31 19:13:57.048971 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.048922 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:13:57.343847 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.343738 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:13:57.513731 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.513694 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:13:57.867426 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.867396 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk"] Jul 31 19:13:57.867789 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.867761 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d82a326c-fa0e-4775-bdb6-d67c7513c1d7" containerName="istio-proxy" Jul 31 19:13:57.867789 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.867771 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d82a326c-fa0e-4775-bdb6-d67c7513c1d7" containerName="istio-proxy" Jul 31 19:13:57.867860 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.867841 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d82a326c-fa0e-4775-bdb6-d67c7513c1d7" containerName="istio-proxy" Jul 31 19:13:57.867860 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.867850 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d82a326c-fa0e-4775-bdb6-d67c7513c1d7" containerName="istio-proxy" Jul 31 19:13:57.870902 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.870886 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:13:57.873427 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.873406 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f3556860-openshift-default-dockercfg-phgpk\"" Jul 31 19:13:57.882918 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.882895 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-workload-socket\") pod \"e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:13:57.883037 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.882933 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-istio-token\") pod \"e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:13:57.883037 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.883012 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-istiod-ca-cert\") pod \"e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:13:57.883145 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.883123 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-istio-podinfo\") pod \"e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:13:57.883191 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.883173 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm5mk\" (UniqueName: \"kubernetes.io/projected/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-kube-api-access-fm5mk\") pod \"e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:13:57.883288 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.883203 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-istio-envoy\") pod \"e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:13:57.883288 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.883258 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-istio-data\") pod \"e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:13:57.883288 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.883285 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-credential-socket\") pod \"e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:13:57.883396 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.883302 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-workload-certs\") pod \"e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:13:57.891571 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.891545 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk"] Jul 31 19:13:57.908368 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.908321 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:13:57.984208 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.984175 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-credential-socket\") pod \"e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:13:57.984208 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.984212 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-workload-certs\") pod \"e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:13:57.984463 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.984282 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-workload-socket\") pod \"e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:13:57.984463 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.984306 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-istio-token\") pod \"e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:13:57.984463 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.984359 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-istiod-ca-cert\") pod \"e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:13:57.984463 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.984435 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-istio-podinfo\") pod \"e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:13:57.984698 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.984476 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm5mk\" (UniqueName: \"kubernetes.io/projected/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-kube-api-access-fm5mk\") pod \"e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:13:57.984698 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.984502 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-istio-envoy\") pod \"e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:13:57.984698 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.984530 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-istio-data\") pod \"e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:13:57.984698 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.984664 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-credential-socket\") pod \"e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:13:57.984899 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.984712 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-workload-certs\") pod \"e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:13:57.984899 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.984856 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-workload-socket\") pod \"e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:13:57.984899 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.984877 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-istio-data\") pod \"e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:13:57.985161 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.985134 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-istiod-ca-cert\") pod \"e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:13:57.986892 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.986866 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-istio-envoy\") pod \"e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:13:57.987116 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.987094 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-istio-podinfo\") pod \"e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:13:57.993720 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.993700 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-istio-token\") pod \"e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:13:57.997564 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:57.997544 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm5mk\" (UniqueName: \"kubernetes.io/projected/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-kube-api-access-fm5mk\") pod \"e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:13:58.123467 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:58.123327 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:13:58.158919 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:58.158883 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:13:58.181525 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:58.181492 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:13:58.321013 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:58.320983 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk"] Jul 31 19:13:58.324082 ip-10-0-139-39 kubenswrapper[2565]: W0731 19:13:58.324057 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb3c24d5_3bc4_4229_bdff_ddcaba3c12e3.slice/crio-4935781cdfcc93d4e6cb753880a66bdf10702f46e8c48f579b6f205d106dd0f3 WatchSource:0}: Error finding container 4935781cdfcc93d4e6cb753880a66bdf10702f46e8c48f579b6f205d106dd0f3: Status 404 returned error can't find the container with id 4935781cdfcc93d4e6cb753880a66bdf10702f46e8c48f579b6f205d106dd0f3 Jul 31 19:13:58.326221 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:58.326190 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:13:58.326327 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:58.326274 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:13:58.326327 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:58.326309 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:13:58.774913 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:58.774874 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" event={"ID":"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3","Type":"ContainerStarted","Data":"240f18dc283588db91a684f6d0c4f5c04c49be68835a583f31036e56a0896cff"} Jul 31 19:13:58.775107 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:58.774957 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" event={"ID":"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3","Type":"ContainerStarted","Data":"4935781cdfcc93d4e6cb753880a66bdf10702f46e8c48f579b6f205d106dd0f3"} Jul 31 19:13:58.795579 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:58.795527 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" podStartSLOduration=1.795510443 podStartE2EDuration="1.795510443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:13:58.794615151 +0000 UTC m=+1815.432604951" watchObservedRunningTime="2026-07-31 19:13:58.795510443 +0000 UTC m=+1815.433500246" Jul 31 19:13:59.182051 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:59.181965 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:13:59.186812 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:59.186790 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:13:59.779528 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:59.779496 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:13:59.780531 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:13:59.780512 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:14:00.168910 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:00.168817 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:14:00.192756 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:00.192718 2565 patch_prober.go:28] interesting pod/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 31 19:14:00.193136 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:00.192786 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" podUID="7e147296-228d-4e0c-a8c3-5998f22f46f9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 31 19:14:00.269327 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:00.269292 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:14:01.287989 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:01.287898 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:14:01.787047 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:01.787004 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:14:01.887615 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:01.887561 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:14:02.287278 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.287253 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:14:02.329039 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.328954 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e147296-228d-4e0c-a8c3-5998f22f46f9-workload-socket\") pod \"7e147296-228d-4e0c-a8c3-5998f22f46f9\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " Jul 31 19:14:02.329039 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.329006 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e147296-228d-4e0c-a8c3-5998f22f46f9-istiod-ca-cert\") pod \"7e147296-228d-4e0c-a8c3-5998f22f46f9\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " Jul 31 19:14:02.329515 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.329065 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e147296-228d-4e0c-a8c3-5998f22f46f9-istio-token\") pod \"7e147296-228d-4e0c-a8c3-5998f22f46f9\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " Jul 31 19:14:02.329515 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.329084 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e147296-228d-4e0c-a8c3-5998f22f46f9-istio-podinfo\") pod \"7e147296-228d-4e0c-a8c3-5998f22f46f9\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " Jul 31 19:14:02.329515 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.329102 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e147296-228d-4e0c-a8c3-5998f22f46f9-credential-socket\") pod \"7e147296-228d-4e0c-a8c3-5998f22f46f9\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " Jul 31 19:14:02.329515 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.329143 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e147296-228d-4e0c-a8c3-5998f22f46f9-istio-envoy\") pod \"7e147296-228d-4e0c-a8c3-5998f22f46f9\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " Jul 31 19:14:02.329515 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.329171 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e147296-228d-4e0c-a8c3-5998f22f46f9-workload-certs\") pod \"7e147296-228d-4e0c-a8c3-5998f22f46f9\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " Jul 31 19:14:02.329515 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.329205 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e147296-228d-4e0c-a8c3-5998f22f46f9-istio-data\") pod \"7e147296-228d-4e0c-a8c3-5998f22f46f9\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " Jul 31 19:14:02.329515 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.329228 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znzt4\" (UniqueName: \"kubernetes.io/projected/7e147296-228d-4e0c-a8c3-5998f22f46f9-kube-api-access-znzt4\") pod \"7e147296-228d-4e0c-a8c3-5998f22f46f9\" (UID: \"7e147296-228d-4e0c-a8c3-5998f22f46f9\") " Jul 31 19:14:02.329515 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.329315 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e147296-228d-4e0c-a8c3-5998f22f46f9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7e147296-228d-4e0c-a8c3-5998f22f46f9" (UID: "7e147296-228d-4e0c-a8c3-5998f22f46f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:14:02.329515 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.329500 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e147296-228d-4e0c-a8c3-5998f22f46f9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7e147296-228d-4e0c-a8c3-5998f22f46f9" (UID: "7e147296-228d-4e0c-a8c3-5998f22f46f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 19:14:02.329993 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.329648 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e147296-228d-4e0c-a8c3-5998f22f46f9-workload-socket\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:14:02.329993 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.329668 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e147296-228d-4e0c-a8c3-5998f22f46f9-istiod-ca-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:14:02.329993 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.329754 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e147296-228d-4e0c-a8c3-5998f22f46f9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7e147296-228d-4e0c-a8c3-5998f22f46f9" (UID: "7e147296-228d-4e0c-a8c3-5998f22f46f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:14:02.329993 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.329951 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e147296-228d-4e0c-a8c3-5998f22f46f9-istio-data" (OuterVolumeSpecName: "istio-data") pod "7e147296-228d-4e0c-a8c3-5998f22f46f9" (UID: "7e147296-228d-4e0c-a8c3-5998f22f46f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:14:02.330256 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.330113 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e147296-228d-4e0c-a8c3-5998f22f46f9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7e147296-228d-4e0c-a8c3-5998f22f46f9" (UID: "7e147296-228d-4e0c-a8c3-5998f22f46f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:14:02.331823 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.331785 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e147296-228d-4e0c-a8c3-5998f22f46f9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7e147296-228d-4e0c-a8c3-5998f22f46f9" (UID: "7e147296-228d-4e0c-a8c3-5998f22f46f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 19:14:02.331961 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.331928 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e147296-228d-4e0c-a8c3-5998f22f46f9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7e147296-228d-4e0c-a8c3-5998f22f46f9" (UID: "7e147296-228d-4e0c-a8c3-5998f22f46f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:14:02.332050 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.332031 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e147296-228d-4e0c-a8c3-5998f22f46f9-istio-token" (OuterVolumeSpecName: "istio-token") pod "7e147296-228d-4e0c-a8c3-5998f22f46f9" (UID: "7e147296-228d-4e0c-a8c3-5998f22f46f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:14:02.332105 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.332040 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e147296-228d-4e0c-a8c3-5998f22f46f9-kube-api-access-znzt4" (OuterVolumeSpecName: "kube-api-access-znzt4") pod "7e147296-228d-4e0c-a8c3-5998f22f46f9" (UID: "7e147296-228d-4e0c-a8c3-5998f22f46f9"). InnerVolumeSpecName "kube-api-access-znzt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:14:02.430280 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.430223 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e147296-228d-4e0c-a8c3-5998f22f46f9-istio-token\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:14:02.430280 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.430276 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e147296-228d-4e0c-a8c3-5998f22f46f9-istio-podinfo\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:14:02.430280 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.430286 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e147296-228d-4e0c-a8c3-5998f22f46f9-credential-socket\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:14:02.430504 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.430294 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e147296-228d-4e0c-a8c3-5998f22f46f9-istio-envoy\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:14:02.430504 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.430302 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e147296-228d-4e0c-a8c3-5998f22f46f9-workload-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:14:02.430504 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.430310 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e147296-228d-4e0c-a8c3-5998f22f46f9-istio-data\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:14:02.430504 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.430317 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znzt4\" (UniqueName: \"kubernetes.io/projected/7e147296-228d-4e0c-a8c3-5998f22f46f9-kube-api-access-znzt4\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:14:02.444594 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.444570 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:14:02.795897 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.795859 2565 generic.go:356] "Generic (PLEG): container finished" podID="7e147296-228d-4e0c-a8c3-5998f22f46f9" containerID="261687edf61e8196ee61950df3cd4c32b7c3751fd9f0638005f58ebf1c569065" exitCode=0 Jul 31 19:14:02.796085 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.795925 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" Jul 31 19:14:02.796085 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.795959 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" event={"ID":"7e147296-228d-4e0c-a8c3-5998f22f46f9","Type":"ContainerDied","Data":"261687edf61e8196ee61950df3cd4c32b7c3751fd9f0638005f58ebf1c569065"} Jul 31 19:14:02.796085 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.796030 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr" event={"ID":"7e147296-228d-4e0c-a8c3-5998f22f46f9","Type":"ContainerDied","Data":"cedf80c555f4d7f42ad004e9758a95e3b2a279c9fdd853558707e80b2479d599"} Jul 31 19:14:02.796085 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.796059 2565 scope.go:117] "RemoveContainer" containerID="261687edf61e8196ee61950df3cd4c32b7c3751fd9f0638005f58ebf1c569065" Jul 31 19:14:02.805706 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.805687 2565 scope.go:117] "RemoveContainer" containerID="23324c75840ed6a649e097db0a30e5a06cf08fc605d3eec04a62424cf077703f" Jul 31 19:14:02.813286 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.813270 2565 scope.go:117] "RemoveContainer" containerID="261687edf61e8196ee61950df3cd4c32b7c3751fd9f0638005f58ebf1c569065" Jul 31 19:14:02.813528 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:14:02.813509 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"261687edf61e8196ee61950df3cd4c32b7c3751fd9f0638005f58ebf1c569065\": container with ID starting with 261687edf61e8196ee61950df3cd4c32b7c3751fd9f0638005f58ebf1c569065 not found: ID does not exist" containerID="261687edf61e8196ee61950df3cd4c32b7c3751fd9f0638005f58ebf1c569065" Jul 31 19:14:02.813571 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.813537 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"261687edf61e8196ee61950df3cd4c32b7c3751fd9f0638005f58ebf1c569065"} err="failed to get container status \"261687edf61e8196ee61950df3cd4c32b7c3751fd9f0638005f58ebf1c569065\": rpc error: code = NotFound desc = could not find container \"261687edf61e8196ee61950df3cd4c32b7c3751fd9f0638005f58ebf1c569065\": container with ID starting with 261687edf61e8196ee61950df3cd4c32b7c3751fd9f0638005f58ebf1c569065 not found: ID does not exist" Jul 31 19:14:02.813571 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.813553 2565 scope.go:117] "RemoveContainer" containerID="23324c75840ed6a649e097db0a30e5a06cf08fc605d3eec04a62424cf077703f" Jul 31 19:14:02.813761 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:14:02.813747 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23324c75840ed6a649e097db0a30e5a06cf08fc605d3eec04a62424cf077703f\": container with ID starting with 23324c75840ed6a649e097db0a30e5a06cf08fc605d3eec04a62424cf077703f not found: ID does not exist" containerID="23324c75840ed6a649e097db0a30e5a06cf08fc605d3eec04a62424cf077703f" Jul 31 19:14:02.813797 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.813768 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23324c75840ed6a649e097db0a30e5a06cf08fc605d3eec04a62424cf077703f"} err="failed to get container status \"23324c75840ed6a649e097db0a30e5a06cf08fc605d3eec04a62424cf077703f\": rpc error: code = NotFound desc = could not find container \"23324c75840ed6a649e097db0a30e5a06cf08fc605d3eec04a62424cf077703f\": container with ID starting with 23324c75840ed6a649e097db0a30e5a06cf08fc605d3eec04a62424cf077703f not found: ID does not exist" Jul 31 19:14:02.823529 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.823504 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr"] Jul 31 19:14:02.832554 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:02.832531 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-b9249e59-gw-openshift-default-5886f46c5b-kbfhr"] Jul 31 19:14:03.886527 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:03.886493 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e147296-228d-4e0c-a8c3-5998f22f46f9" path="/var/lib/kubelet/pods/7e147296-228d-4e0c-a8c3-5998f22f46f9/volumes" Jul 31 19:14:04.256029 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:04.255989 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:14:13.869827 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:13.869795 2565 generic.go:356] "Generic (PLEG): container finished" podID="db3c24d5-3bc4-4229-bdff-ddcaba3c12e3" containerID="240f18dc283588db91a684f6d0c4f5c04c49be68835a583f31036e56a0896cff" exitCode=0 Jul 31 19:14:13.870216 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:13.869871 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" event={"ID":"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3","Type":"ContainerDied","Data":"240f18dc283588db91a684f6d0c4f5c04c49be68835a583f31036e56a0896cff"} Jul 31 19:14:13.870424 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:13.870407 2565 scope.go:117] "RemoveContainer" containerID="240f18dc283588db91a684f6d0c4f5c04c49be68835a583f31036e56a0896cff" Jul 31 19:14:13.871560 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:13.871532 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:14:13.871631 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:13.871599 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:14:13.871672 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:13.871636 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:14:14.779732 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:14.779695 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:14:14.876787 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:14.876745 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" event={"ID":"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3","Type":"ContainerStarted","Data":"04e29122df6970b26c8a5a615f95b672a70c3f668d17709a93aa07bea80d4351"} Jul 31 19:14:15.182624 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:15.182534 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:14:15.183718 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:15.183682 2565 patch_prober.go:28] interesting pod/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 31 19:14:15.183872 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:15.183740 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" podUID="db3c24d5-3bc4-4229-bdff-ddcaba3c12e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 31 19:14:16.182027 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:16.181996 2565 patch_prober.go:28] interesting pod/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 31 19:14:16.182420 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:16.182050 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" podUID="db3c24d5-3bc4-4229-bdff-ddcaba3c12e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 31 19:14:17.182815 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:17.182776 2565 patch_prober.go:28] interesting pod/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 31 19:14:17.183232 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:17.182833 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" podUID="db3c24d5-3bc4-4229-bdff-ddcaba3c12e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 31 19:14:18.181882 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:18.181843 2565 patch_prober.go:28] interesting pod/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 31 19:14:18.182064 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:18.181915 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" podUID="db3c24d5-3bc4-4229-bdff-ddcaba3c12e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 31 19:14:19.182150 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:19.182116 2565 patch_prober.go:28] interesting pod/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 31 19:14:19.182561 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:19.182170 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" podUID="db3c24d5-3bc4-4229-bdff-ddcaba3c12e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 31 19:14:20.186187 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:20.186156 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:14:20.186596 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:20.186543 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:14:20.187221 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:20.187201 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:14:25.949541 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:25.949506 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3556860-ptk4b"] Jul 31 19:14:25.949912 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:25.949861 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d82a326c-fa0e-4775-bdb6-d67c7513c1d7" containerName="istio-proxy" Jul 31 19:14:25.949912 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:25.949871 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d82a326c-fa0e-4775-bdb6-d67c7513c1d7" containerName="istio-proxy" Jul 31 19:14:25.949912 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:25.949888 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e147296-228d-4e0c-a8c3-5998f22f46f9" containerName="istio-proxy" Jul 31 19:14:25.949912 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:25.949902 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e147296-228d-4e0c-a8c3-5998f22f46f9" containerName="istio-proxy" Jul 31 19:14:25.950046 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:25.949915 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e147296-228d-4e0c-a8c3-5998f22f46f9" containerName="istio-proxy" Jul 31 19:14:25.950046 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:25.949921 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e147296-228d-4e0c-a8c3-5998f22f46f9" containerName="istio-proxy" Jul 31 19:14:25.950046 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:25.949982 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e147296-228d-4e0c-a8c3-5998f22f46f9" containerName="istio-proxy" Jul 31 19:14:25.952999 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:25.952983 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3556860-ptk4b" Jul 31 19:14:25.955282 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:25.955264 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 31 19:14:25.955346 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:25.955264 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4qkh\"" Jul 31 19:14:25.962398 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:25.962374 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3556860-ptk4b"] Jul 31 19:14:26.059710 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:26.059680 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb8b643-766c-4070-bedb-7f3e1cf70aa4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f3556860-ptk4b\" (UID: \"2bb8b643-766c-4070-bedb-7f3e1cf70aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3556860-ptk4b" Jul 31 19:14:26.160975 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:26.160937 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb8b643-766c-4070-bedb-7f3e1cf70aa4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f3556860-ptk4b\" (UID: \"2bb8b643-766c-4070-bedb-7f3e1cf70aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3556860-ptk4b" Jul 31 19:14:26.161526 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:26.161506 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb8b643-766c-4070-bedb-7f3e1cf70aa4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f3556860-ptk4b\" (UID: \"2bb8b643-766c-4070-bedb-7f3e1cf70aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3556860-ptk4b" Jul 31 19:14:26.264644 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:26.264609 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3556860-ptk4b" Jul 31 19:14:26.399506 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:26.399481 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3556860-ptk4b"] Jul 31 19:14:26.401422 ip-10-0-139-39 kubenswrapper[2565]: W0731 19:14:26.401394 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb8b643_766c_4070_bedb_7f3e1cf70aa4.slice/crio-b17efd91bef30bc4317ec57cc7febb8145516e0d832ca3f977d3fe965548abdf WatchSource:0}: Error finding container b17efd91bef30bc4317ec57cc7febb8145516e0d832ca3f977d3fe965548abdf: Status 404 returned error can't find the container with id b17efd91bef30bc4317ec57cc7febb8145516e0d832ca3f977d3fe965548abdf Jul 31 19:14:26.403325 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:26.403307 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 19:14:26.934277 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:26.934217 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3556860-ptk4b" event={"ID":"2bb8b643-766c-4070-bedb-7f3e1cf70aa4","Type":"ContainerStarted","Data":"b17efd91bef30bc4317ec57cc7febb8145516e0d832ca3f977d3fe965548abdf"} Jul 31 19:14:29.950893 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:29.950854 2565 generic.go:356] "Generic (PLEG): container finished" podID="2bb8b643-766c-4070-bedb-7f3e1cf70aa4" containerID="aed4dbc8d371515e0681669b261f48df4b778284ee6cd0e60f77bdbeb4dc6f60" exitCode=0 Jul 31 19:14:29.951329 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:29.950937 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3556860-ptk4b" event={"ID":"2bb8b643-766c-4070-bedb-7f3e1cf70aa4","Type":"ContainerDied","Data":"aed4dbc8d371515e0681669b261f48df4b778284ee6cd0e60f77bdbeb4dc6f60"} Jul 31 19:14:31.092777 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:31.092751 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3556860-ptk4b" Jul 31 19:14:31.210731 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:31.210634 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb8b643-766c-4070-bedb-7f3e1cf70aa4-maas-api-service-ca\") pod \"2bb8b643-766c-4070-bedb-7f3e1cf70aa4\" (UID: \"2bb8b643-766c-4070-bedb-7f3e1cf70aa4\") " Jul 31 19:14:31.211018 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:31.210993 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bb8b643-766c-4070-bedb-7f3e1cf70aa4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2bb8b643-766c-4070-bedb-7f3e1cf70aa4" (UID: "2bb8b643-766c-4070-bedb-7f3e1cf70aa4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 19:14:31.312271 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:31.312209 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb8b643-766c-4070-bedb-7f3e1cf70aa4-maas-api-service-ca\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:14:31.962354 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:31.962327 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3556860-ptk4b" Jul 31 19:14:31.962520 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:31.962368 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3556860-ptk4b" event={"ID":"2bb8b643-766c-4070-bedb-7f3e1cf70aa4","Type":"ContainerDied","Data":"b17efd91bef30bc4317ec57cc7febb8145516e0d832ca3f977d3fe965548abdf"} Jul 31 19:14:31.962520 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:31.962416 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b17efd91bef30bc4317ec57cc7febb8145516e0d832ca3f977d3fe965548abdf" Jul 31 19:14:42.213315 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:42.213269 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3556860-ptk4b"] Jul 31 19:14:42.217279 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:42.217253 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3556860-ptk4b"] Jul 31 19:14:42.341478 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:42.341433 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:14:42.407953 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:42.407910 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:14:43.393584 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:43.393541 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk"] Jul 31 19:14:43.394050 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:43.393763 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" podUID="db3c24d5-3bc4-4229-bdff-ddcaba3c12e3" containerName="istio-proxy" containerID="cri-o://04e29122df6970b26c8a5a615f95b672a70c3f668d17709a93aa07bea80d4351" gracePeriod=30 Jul 31 19:14:43.680280 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:43.680136 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:14:43.741937 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:43.741906 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:14:43.884835 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:43.884798 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:14:43.888691 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:43.888660 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bb8b643-766c-4070-bedb-7f3e1cf70aa4" path="/var/lib/kubelet/pods/2bb8b643-766c-4070-bedb-7f3e1cf70aa4/volumes" Jul 31 19:14:44.154787 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.154748 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8"] Jul 31 19:14:44.155201 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.155183 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bb8b643-766c-4070-bedb-7f3e1cf70aa4" containerName="revoke-keys" Jul 31 19:14:44.155316 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.155204 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bb8b643-766c-4070-bedb-7f3e1cf70aa4" containerName="revoke-keys" Jul 31 19:14:44.155316 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.155311 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e147296-228d-4e0c-a8c3-5998f22f46f9" containerName="istio-proxy" Jul 31 19:14:44.155422 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.155330 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bb8b643-766c-4070-bedb-7f3e1cf70aa4" containerName="revoke-keys" Jul 31 19:14:44.159996 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.159973 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:44.163963 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.163944 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-bc42f1be-openshift-default-dockercfg-kqtg9\"" Jul 31 19:14:44.173701 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.173680 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8"] Jul 31 19:14:44.184360 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.184334 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:14:44.232329 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.232298 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a337efd3-fd30-4567-866a-72023cd150b0-workload-certs\") pod \"e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:44.232329 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.232334 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgwm7\" (UniqueName: \"kubernetes.io/projected/a337efd3-fd30-4567-866a-72023cd150b0-kube-api-access-jgwm7\") pod \"e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:44.232570 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.232369 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a337efd3-fd30-4567-866a-72023cd150b0-credential-socket\") pod \"e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:44.232570 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.232424 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a337efd3-fd30-4567-866a-72023cd150b0-istio-podinfo\") pod \"e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:44.232570 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.232451 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a337efd3-fd30-4567-866a-72023cd150b0-workload-socket\") pod \"e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:44.232570 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.232528 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a337efd3-fd30-4567-866a-72023cd150b0-istio-envoy\") pod \"e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:44.232570 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.232567 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a337efd3-fd30-4567-866a-72023cd150b0-istiod-ca-cert\") pod \"e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:44.232768 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.232619 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a337efd3-fd30-4567-866a-72023cd150b0-istio-data\") pod \"e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:44.232768 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.232683 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a337efd3-fd30-4567-866a-72023cd150b0-istio-token\") pod \"e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:44.333740 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.333648 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a337efd3-fd30-4567-866a-72023cd150b0-credential-socket\") pod \"e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:44.333740 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.333702 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a337efd3-fd30-4567-866a-72023cd150b0-istio-podinfo\") pod \"e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:44.333740 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.333720 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a337efd3-fd30-4567-866a-72023cd150b0-workload-socket\") pod \"e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:44.334034 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.333744 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a337efd3-fd30-4567-866a-72023cd150b0-istio-envoy\") pod \"e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:44.334034 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.333762 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a337efd3-fd30-4567-866a-72023cd150b0-istiod-ca-cert\") pod \"e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:44.334034 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.333785 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a337efd3-fd30-4567-866a-72023cd150b0-istio-data\") pod \"e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:44.334034 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.333843 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a337efd3-fd30-4567-866a-72023cd150b0-istio-token\") pod \"e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:44.334034 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.333904 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a337efd3-fd30-4567-866a-72023cd150b0-workload-certs\") pod \"e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:44.334034 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.333926 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgwm7\" (UniqueName: \"kubernetes.io/projected/a337efd3-fd30-4567-866a-72023cd150b0-kube-api-access-jgwm7\") pod \"e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:44.334332 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.334142 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a337efd3-fd30-4567-866a-72023cd150b0-credential-socket\") pod \"e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:44.334332 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.334313 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a337efd3-fd30-4567-866a-72023cd150b0-istio-data\") pod \"e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:44.334411 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.334387 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a337efd3-fd30-4567-866a-72023cd150b0-workload-certs\") pod \"e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:44.334643 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.334619 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a337efd3-fd30-4567-866a-72023cd150b0-istiod-ca-cert\") pod \"e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:44.334753 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.334732 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a337efd3-fd30-4567-866a-72023cd150b0-workload-socket\") pod \"e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:44.336128 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.336102 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a337efd3-fd30-4567-866a-72023cd150b0-istio-envoy\") pod \"e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:44.336375 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.336356 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a337efd3-fd30-4567-866a-72023cd150b0-istio-podinfo\") pod \"e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:44.342468 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.342447 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a337efd3-fd30-4567-866a-72023cd150b0-istio-token\") pod \"e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:44.342654 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.342634 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgwm7\" (UniqueName: \"kubernetes.io/projected/a337efd3-fd30-4567-866a-72023cd150b0-kube-api-access-jgwm7\") pod \"e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:44.470544 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.470500 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:44.526994 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.526902 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:14:44.613379 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.613356 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8"] Jul 31 19:14:44.615419 ip-10-0-139-39 kubenswrapper[2565]: W0731 19:14:44.615391 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda337efd3_fd30_4567_866a_72023cd150b0.slice/crio-754c3d60042c8459e2bd3f3d39ed78a888c9e0ef72b0d8743faab00ea8550860 WatchSource:0}: Error finding container 754c3d60042c8459e2bd3f3d39ed78a888c9e0ef72b0d8743faab00ea8550860: Status 404 returned error can't find the container with id 754c3d60042c8459e2bd3f3d39ed78a888c9e0ef72b0d8743faab00ea8550860 Jul 31 19:14:44.617595 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.617561 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:14:44.617696 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.617637 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:14:44.617759 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.617706 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:14:44.872034 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:44.871942 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:14:45.025139 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:45.025104 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" event={"ID":"a337efd3-fd30-4567-866a-72023cd150b0","Type":"ContainerStarted","Data":"db7733db16c83f3a4b56ab0082fb1b5a42c871e7a0b9d9e70b5caf09f581ffdf"} Jul 31 19:14:45.025331 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:45.025161 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" event={"ID":"a337efd3-fd30-4567-866a-72023cd150b0","Type":"ContainerStarted","Data":"754c3d60042c8459e2bd3f3d39ed78a888c9e0ef72b0d8743faab00ea8550860"} Jul 31 19:14:45.049980 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:45.049929 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" podStartSLOduration=1.049914952 podStartE2EDuration="1.049914952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:14:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:14:45.047740665 +0000 UTC m=+1861.685730466" watchObservedRunningTime="2026-07-31 19:14:45.049914952 +0000 UTC m=+1861.687904751" Jul 31 19:14:45.471293 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:45.471230 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:45.476179 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:45.476152 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:46.030369 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:46.030332 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:46.031212 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:46.031194 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:14:46.450104 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:46.450022 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:14:46.546791 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:46.546754 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:14:47.555445 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:47.555412 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:14:48.013967 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:48.013926 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:14:48.195537 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:48.195497 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:14:48.295856 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:48.295762 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:14:48.673805 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:48.673781 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:14:48.776367 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:48.776326 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-istiod-ca-cert\") pod \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " Jul 31 19:14:48.776367 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:48.776372 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-istio-envoy\") pod \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " Jul 31 19:14:48.776616 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:48.776456 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-workload-socket\") pod \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " Jul 31 19:14:48.776616 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:48.776482 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fm5mk\" (UniqueName: \"kubernetes.io/projected/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-kube-api-access-fm5mk\") pod \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " Jul 31 19:14:48.776616 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:48.776509 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-workload-certs\") pod \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " Jul 31 19:14:48.776616 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:48.776542 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-istio-data\") pod \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " Jul 31 19:14:48.776616 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:48.776609 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-istio-podinfo\") pod \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " Jul 31 19:14:48.776888 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:48.776656 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-istio-token\") pod \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " Jul 31 19:14:48.776888 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:48.776705 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-credential-socket\") pod \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\" (UID: \"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3\") " Jul 31 19:14:48.776983 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:48.776700 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "db3c24d5-3bc4-4229-bdff-ddcaba3c12e3" (UID: "db3c24d5-3bc4-4229-bdff-ddcaba3c12e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 19:14:48.777039 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:48.776806 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "db3c24d5-3bc4-4229-bdff-ddcaba3c12e3" (UID: "db3c24d5-3bc4-4229-bdff-ddcaba3c12e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:14:48.777039 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:48.776816 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "db3c24d5-3bc4-4229-bdff-ddcaba3c12e3" (UID: "db3c24d5-3bc4-4229-bdff-ddcaba3c12e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:14:48.777039 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:48.776874 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-istio-data" (OuterVolumeSpecName: "istio-data") pod "db3c24d5-3bc4-4229-bdff-ddcaba3c12e3" (UID: "db3c24d5-3bc4-4229-bdff-ddcaba3c12e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:14:48.777039 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:48.776974 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "db3c24d5-3bc4-4229-bdff-ddcaba3c12e3" (UID: "db3c24d5-3bc4-4229-bdff-ddcaba3c12e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:14:48.777280 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:48.777132 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-istiod-ca-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:14:48.777280 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:48.777154 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-workload-socket\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:14:48.777280 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:48.777168 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-workload-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:14:48.777280 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:48.777182 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-istio-data\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:14:48.777280 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:48.777194 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-credential-socket\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:14:48.778868 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:48.778841 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "db3c24d5-3bc4-4229-bdff-ddcaba3c12e3" (UID: "db3c24d5-3bc4-4229-bdff-ddcaba3c12e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 19:14:48.778868 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:48.778864 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-istio-token" (OuterVolumeSpecName: "istio-token") pod "db3c24d5-3bc4-4229-bdff-ddcaba3c12e3" (UID: "db3c24d5-3bc4-4229-bdff-ddcaba3c12e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:14:48.779065 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:48.779051 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "db3c24d5-3bc4-4229-bdff-ddcaba3c12e3" (UID: "db3c24d5-3bc4-4229-bdff-ddcaba3c12e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:14:48.779124 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:48.779104 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-kube-api-access-fm5mk" (OuterVolumeSpecName: "kube-api-access-fm5mk") pod "db3c24d5-3bc4-4229-bdff-ddcaba3c12e3" (UID: "db3c24d5-3bc4-4229-bdff-ddcaba3c12e3"). InnerVolumeSpecName "kube-api-access-fm5mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:14:48.877829 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:48.877742 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm5mk\" (UniqueName: \"kubernetes.io/projected/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-kube-api-access-fm5mk\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:14:48.877829 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:48.877772 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-istio-podinfo\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:14:48.877829 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:48.877781 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-istio-token\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:14:48.877829 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:48.877790 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3-istio-envoy\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:14:49.047112 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:49.047074 2565 generic.go:356] "Generic (PLEG): container finished" podID="db3c24d5-3bc4-4229-bdff-ddcaba3c12e3" containerID="04e29122df6970b26c8a5a615f95b672a70c3f668d17709a93aa07bea80d4351" exitCode=0 Jul 31 19:14:49.047310 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:49.047122 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" Jul 31 19:14:49.047310 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:49.047165 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" event={"ID":"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3","Type":"ContainerDied","Data":"04e29122df6970b26c8a5a615f95b672a70c3f668d17709a93aa07bea80d4351"} Jul 31 19:14:49.047310 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:49.047222 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk" event={"ID":"db3c24d5-3bc4-4229-bdff-ddcaba3c12e3","Type":"ContainerDied","Data":"4935781cdfcc93d4e6cb753880a66bdf10702f46e8c48f579b6f205d106dd0f3"} Jul 31 19:14:49.047310 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:49.047275 2565 scope.go:117] "RemoveContainer" containerID="04e29122df6970b26c8a5a615f95b672a70c3f668d17709a93aa07bea80d4351" Jul 31 19:14:49.056606 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:49.056589 2565 scope.go:117] "RemoveContainer" containerID="240f18dc283588db91a684f6d0c4f5c04c49be68835a583f31036e56a0896cff" Jul 31 19:14:49.064915 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:49.064898 2565 scope.go:117] "RemoveContainer" containerID="04e29122df6970b26c8a5a615f95b672a70c3f668d17709a93aa07bea80d4351" Jul 31 19:14:49.065144 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:14:49.065129 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04e29122df6970b26c8a5a615f95b672a70c3f668d17709a93aa07bea80d4351\": container with ID starting with 04e29122df6970b26c8a5a615f95b672a70c3f668d17709a93aa07bea80d4351 not found: ID does not exist" containerID="04e29122df6970b26c8a5a615f95b672a70c3f668d17709a93aa07bea80d4351" Jul 31 19:14:49.065183 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:49.065150 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04e29122df6970b26c8a5a615f95b672a70c3f668d17709a93aa07bea80d4351"} err="failed to get container status \"04e29122df6970b26c8a5a615f95b672a70c3f668d17709a93aa07bea80d4351\": rpc error: code = NotFound desc = could not find container \"04e29122df6970b26c8a5a615f95b672a70c3f668d17709a93aa07bea80d4351\": container with ID starting with 04e29122df6970b26c8a5a615f95b672a70c3f668d17709a93aa07bea80d4351 not found: ID does not exist" Jul 31 19:14:49.065183 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:49.065165 2565 scope.go:117] "RemoveContainer" containerID="240f18dc283588db91a684f6d0c4f5c04c49be68835a583f31036e56a0896cff" Jul 31 19:14:49.065390 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:14:49.065373 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"240f18dc283588db91a684f6d0c4f5c04c49be68835a583f31036e56a0896cff\": container with ID starting with 240f18dc283588db91a684f6d0c4f5c04c49be68835a583f31036e56a0896cff not found: ID does not exist" containerID="240f18dc283588db91a684f6d0c4f5c04c49be68835a583f31036e56a0896cff" Jul 31 19:14:49.065437 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:49.065396 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"240f18dc283588db91a684f6d0c4f5c04c49be68835a583f31036e56a0896cff"} err="failed to get container status \"240f18dc283588db91a684f6d0c4f5c04c49be68835a583f31036e56a0896cff\": rpc error: code = NotFound desc = could not find container \"240f18dc283588db91a684f6d0c4f5c04c49be68835a583f31036e56a0896cff\": container with ID starting with 240f18dc283588db91a684f6d0c4f5c04c49be68835a583f31036e56a0896cff not found: ID does not exist" Jul 31 19:14:49.070227 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:49.070204 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk"] Jul 31 19:14:49.075307 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:49.075283 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f3556860-openshift-default-5c646bb74-pgtxk"] Jul 31 19:14:49.887691 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:49.887658 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db3c24d5-3bc4-4229-bdff-ddcaba3c12e3" path="/var/lib/kubelet/pods/db3c24d5-3bc4-4229-bdff-ddcaba3c12e3/volumes" Jul 31 19:14:51.054138 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:14:51.054108 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:15:00.151557 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:00.151525 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bc42f1be-29758755-w62w6"] Jul 31 19:15:00.151909 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:00.151888 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db3c24d5-3bc4-4229-bdff-ddcaba3c12e3" containerName="istio-proxy" Jul 31 19:15:00.151909 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:00.151898 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="db3c24d5-3bc4-4229-bdff-ddcaba3c12e3" containerName="istio-proxy" Jul 31 19:15:00.151988 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:00.151919 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db3c24d5-3bc4-4229-bdff-ddcaba3c12e3" containerName="istio-proxy" Jul 31 19:15:00.151988 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:00.151925 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="db3c24d5-3bc4-4229-bdff-ddcaba3c12e3" containerName="istio-proxy" Jul 31 19:15:00.152052 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:00.152001 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="db3c24d5-3bc4-4229-bdff-ddcaba3c12e3" containerName="istio-proxy" Jul 31 19:15:00.156322 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:00.156305 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bc42f1be-29758755-w62w6" Jul 31 19:15:00.158992 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:00.158971 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4qkh\"" Jul 31 19:15:00.167209 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:00.167188 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bc42f1be-29758755-w62w6"] Jul 31 19:15:00.168394 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:00.168368 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bc42f1be-29758755-w62w6" Jul 31 19:15:00.520384 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:00.520358 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bc42f1be-29758755-w62w6"] Jul 31 19:15:00.522506 ip-10-0-139-39 kubenswrapper[2565]: W0731 19:15:00.522469 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podece0d223_3735_4994_97c7_f6c66863b26b.slice/crio-14a1ede0007abd7b4f4d8eb1fd7b9d3451e2890f35debd76d32056e09fff1175 WatchSource:0}: Error finding container 14a1ede0007abd7b4f4d8eb1fd7b9d3451e2890f35debd76d32056e09fff1175: Status 404 returned error can't find the container with id 14a1ede0007abd7b4f4d8eb1fd7b9d3451e2890f35debd76d32056e09fff1175 Jul 31 19:15:01.031307 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:01.031269 2565 patch_prober.go:28] interesting pod/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 31 19:15:01.031536 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:01.031333 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" podUID="a337efd3-fd30-4567-866a-72023cd150b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 31 19:15:01.113388 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:01.113301 2565 generic.go:356] "Generic (PLEG): container finished" podID="a337efd3-fd30-4567-866a-72023cd150b0" containerID="db7733db16c83f3a4b56ab0082fb1b5a42c871e7a0b9d9e70b5caf09f581ffdf" exitCode=0 Jul 31 19:15:01.113388 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:01.113376 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" event={"ID":"a337efd3-fd30-4567-866a-72023cd150b0","Type":"ContainerDied","Data":"db7733db16c83f3a4b56ab0082fb1b5a42c871e7a0b9d9e70b5caf09f581ffdf"} Jul 31 19:15:01.113831 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:01.113805 2565 scope.go:117] "RemoveContainer" containerID="db7733db16c83f3a4b56ab0082fb1b5a42c871e7a0b9d9e70b5caf09f581ffdf" Jul 31 19:15:01.114980 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:01.114944 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:15:01.115085 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:01.115009 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bc42f1be-29758755-w62w6" event={"ID":"ece0d223-3735-4994-97c7-f6c66863b26b","Type":"ContainerStarted","Data":"55655d51572cf8bfef4006242bfc0b5733d98137026e62ad9e0060bae0ec2ec5"} Jul 31 19:15:01.115085 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:01.115028 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:15:01.115085 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:01.115049 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bc42f1be-29758755-w62w6" event={"ID":"ece0d223-3735-4994-97c7-f6c66863b26b","Type":"ContainerStarted","Data":"14a1ede0007abd7b4f4d8eb1fd7b9d3451e2890f35debd76d32056e09fff1175"} Jul 31 19:15:01.115085 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:01.115074 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:15:01.160013 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:01.159965 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bc42f1be-29758755-w62w6" podStartSLOduration=1.159948631 podStartE2EDuration="1.159948631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:15:01.157341865 +0000 UTC m=+1877.795331666" watchObservedRunningTime="2026-07-31 19:15:01.159948631 +0000 UTC m=+1877.797938431" Jul 31 19:15:02.122066 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:02.122033 2565 generic.go:356] "Generic (PLEG): container finished" podID="ece0d223-3735-4994-97c7-f6c66863b26b" containerID="55655d51572cf8bfef4006242bfc0b5733d98137026e62ad9e0060bae0ec2ec5" exitCode=0 Jul 31 19:15:02.122297 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:02.122117 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bc42f1be-29758755-w62w6" event={"ID":"ece0d223-3735-4994-97c7-f6c66863b26b","Type":"ContainerDied","Data":"55655d51572cf8bfef4006242bfc0b5733d98137026e62ad9e0060bae0ec2ec5"} Jul 31 19:15:02.124464 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:02.124439 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" event={"ID":"a337efd3-fd30-4567-866a-72023cd150b0","Type":"ContainerStarted","Data":"c1d3bca1afb08e190256d34860b61fe24484d416b46841ec446d482247264841"} Jul 31 19:15:02.471713 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:02.471619 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:15:02.472078 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:02.471872 2565 patch_prober.go:28] interesting pod/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 31 19:15:02.472078 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:02.471923 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" podUID="a337efd3-fd30-4567-866a-72023cd150b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 31 19:15:03.311580 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:03.311548 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bc42f1be-29758755-w62w6" Jul 31 19:15:03.471896 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:03.471815 2565 patch_prober.go:28] interesting pod/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 31 19:15:03.471896 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:03.471875 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" podUID="a337efd3-fd30-4567-866a-72023cd150b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 31 19:15:04.151795 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:04.151686 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bc42f1be-29758755-w62w6" event={"ID":"ece0d223-3735-4994-97c7-f6c66863b26b","Type":"ContainerDied","Data":"14a1ede0007abd7b4f4d8eb1fd7b9d3451e2890f35debd76d32056e09fff1175"} Jul 31 19:15:04.151966 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:04.151792 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14a1ede0007abd7b4f4d8eb1fd7b9d3451e2890f35debd76d32056e09fff1175" Jul 31 19:15:04.151966 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:04.151710 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bc42f1be-29758755-w62w6" Jul 31 19:15:04.471027 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:04.470932 2565 patch_prober.go:28] interesting pod/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 31 19:15:04.471027 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:04.470992 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" podUID="a337efd3-fd30-4567-866a-72023cd150b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 31 19:15:05.471167 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:05.471125 2565 patch_prober.go:28] interesting pod/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 31 19:15:05.471617 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:05.471190 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" podUID="a337efd3-fd30-4567-866a-72023cd150b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 31 19:15:06.471395 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:06.471310 2565 patch_prober.go:28] interesting pod/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 31 19:15:06.471395 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:06.471369 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" podUID="a337efd3-fd30-4567-866a-72023cd150b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 31 19:15:07.474639 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:07.474608 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:15:07.475013 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:07.474871 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:15:07.475446 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:07.475431 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:15:18.087032 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:18.086995 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bc42f1be-29758755-w62w6"] Jul 31 19:15:18.090327 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:18.090302 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bc42f1be-29758755-w62w6"] Jul 31 19:15:18.091802 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:18.091776 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:15:18.166437 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:18.166398 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:15:18.233804 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:18.233770 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:15:18.406678 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:18.406563 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8"] Jul 31 19:15:18.407137 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:18.407102 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" podUID="a337efd3-fd30-4567-866a-72023cd150b0" containerName="istio-proxy" containerID="cri-o://c1d3bca1afb08e190256d34860b61fe24484d416b46841ec446d482247264841" gracePeriod=30 Jul 31 19:15:18.712391 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:18.712315 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:15:18.792774 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:18.792742 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:15:19.144963 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:19.144932 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:15:19.245919 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:19.245887 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:15:19.344443 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:19.344403 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:15:19.886460 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:19.886422 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ece0d223-3735-4994-97c7-f6c66863b26b" path="/var/lib/kubelet/pods/ece0d223-3735-4994-97c7-f6c66863b26b/volumes" Jul 31 19:15:20.859058 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:20.859021 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:15:22.477545 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:22.477500 2565 patch_prober.go:28] interesting pod/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 31 19:15:22.478070 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:22.477566 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" podUID="a337efd3-fd30-4567-866a-72023cd150b0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 31 19:15:23.670271 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:23.670225 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:15:23.763500 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:23.763472 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgwm7\" (UniqueName: \"kubernetes.io/projected/a337efd3-fd30-4567-866a-72023cd150b0-kube-api-access-jgwm7\") pod \"a337efd3-fd30-4567-866a-72023cd150b0\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " Jul 31 19:15:23.763648 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:23.763506 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a337efd3-fd30-4567-866a-72023cd150b0-workload-certs\") pod \"a337efd3-fd30-4567-866a-72023cd150b0\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " Jul 31 19:15:23.763648 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:23.763526 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a337efd3-fd30-4567-866a-72023cd150b0-istiod-ca-cert\") pod \"a337efd3-fd30-4567-866a-72023cd150b0\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " Jul 31 19:15:23.763648 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:23.763553 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a337efd3-fd30-4567-866a-72023cd150b0-workload-socket\") pod \"a337efd3-fd30-4567-866a-72023cd150b0\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " Jul 31 19:15:23.763648 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:23.763610 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a337efd3-fd30-4567-866a-72023cd150b0-istio-envoy\") pod \"a337efd3-fd30-4567-866a-72023cd150b0\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " Jul 31 19:15:23.763843 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:23.763676 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a337efd3-fd30-4567-866a-72023cd150b0-istio-data\") pod \"a337efd3-fd30-4567-866a-72023cd150b0\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " Jul 31 19:15:23.763843 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:23.763711 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a337efd3-fd30-4567-866a-72023cd150b0-credential-socket\") pod \"a337efd3-fd30-4567-866a-72023cd150b0\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " Jul 31 19:15:23.763843 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:23.763761 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a337efd3-fd30-4567-866a-72023cd150b0-istio-podinfo\") pod \"a337efd3-fd30-4567-866a-72023cd150b0\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " Jul 31 19:15:23.763843 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:23.763780 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a337efd3-fd30-4567-866a-72023cd150b0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a337efd3-fd30-4567-866a-72023cd150b0" (UID: "a337efd3-fd30-4567-866a-72023cd150b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:15:23.763843 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:23.763804 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a337efd3-fd30-4567-866a-72023cd150b0-istio-token\") pod \"a337efd3-fd30-4567-866a-72023cd150b0\" (UID: \"a337efd3-fd30-4567-866a-72023cd150b0\") " Jul 31 19:15:23.764105 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:23.763958 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a337efd3-fd30-4567-866a-72023cd150b0-istio-data" (OuterVolumeSpecName: "istio-data") pod "a337efd3-fd30-4567-866a-72023cd150b0" (UID: "a337efd3-fd30-4567-866a-72023cd150b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:15:23.764105 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:23.763987 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a337efd3-fd30-4567-866a-72023cd150b0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a337efd3-fd30-4567-866a-72023cd150b0" (UID: "a337efd3-fd30-4567-866a-72023cd150b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:15:23.764105 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:23.764072 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a337efd3-fd30-4567-866a-72023cd150b0-workload-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:15:23.764281 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:23.764111 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a337efd3-fd30-4567-866a-72023cd150b0-istio-data\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:15:23.764281 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:23.764125 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a337efd3-fd30-4567-866a-72023cd150b0-credential-socket\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:15:23.764281 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:23.764140 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a337efd3-fd30-4567-866a-72023cd150b0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a337efd3-fd30-4567-866a-72023cd150b0" (UID: "a337efd3-fd30-4567-866a-72023cd150b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:15:23.764281 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:23.764150 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a337efd3-fd30-4567-866a-72023cd150b0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a337efd3-fd30-4567-866a-72023cd150b0" (UID: "a337efd3-fd30-4567-866a-72023cd150b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 19:15:23.765852 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:23.765820 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a337efd3-fd30-4567-866a-72023cd150b0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a337efd3-fd30-4567-866a-72023cd150b0" (UID: "a337efd3-fd30-4567-866a-72023cd150b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:15:23.765952 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:23.765930 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a337efd3-fd30-4567-866a-72023cd150b0-istio-token" (OuterVolumeSpecName: "istio-token") pod "a337efd3-fd30-4567-866a-72023cd150b0" (UID: "a337efd3-fd30-4567-866a-72023cd150b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:15:23.766146 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:23.766124 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a337efd3-fd30-4567-866a-72023cd150b0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a337efd3-fd30-4567-866a-72023cd150b0" (UID: "a337efd3-fd30-4567-866a-72023cd150b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 19:15:23.766221 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:23.766204 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a337efd3-fd30-4567-866a-72023cd150b0-kube-api-access-jgwm7" (OuterVolumeSpecName: "kube-api-access-jgwm7") pod "a337efd3-fd30-4567-866a-72023cd150b0" (UID: "a337efd3-fd30-4567-866a-72023cd150b0"). InnerVolumeSpecName "kube-api-access-jgwm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:15:23.865445 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:23.865410 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a337efd3-fd30-4567-866a-72023cd150b0-istio-envoy\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:15:23.865445 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:23.865439 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a337efd3-fd30-4567-866a-72023cd150b0-istio-podinfo\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:15:23.865445 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:23.865448 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a337efd3-fd30-4567-866a-72023cd150b0-istio-token\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:15:23.865676 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:23.865456 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgwm7\" (UniqueName: \"kubernetes.io/projected/a337efd3-fd30-4567-866a-72023cd150b0-kube-api-access-jgwm7\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:15:23.865676 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:23.865467 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a337efd3-fd30-4567-866a-72023cd150b0-istiod-ca-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:15:23.865676 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:23.865475 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a337efd3-fd30-4567-866a-72023cd150b0-workload-socket\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:15:24.250041 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:24.250004 2565 generic.go:356] "Generic (PLEG): container finished" podID="a337efd3-fd30-4567-866a-72023cd150b0" containerID="c1d3bca1afb08e190256d34860b61fe24484d416b46841ec446d482247264841" exitCode=0 Jul 31 19:15:24.250195 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:24.250058 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" Jul 31 19:15:24.250195 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:24.250096 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" event={"ID":"a337efd3-fd30-4567-866a-72023cd150b0","Type":"ContainerDied","Data":"c1d3bca1afb08e190256d34860b61fe24484d416b46841ec446d482247264841"} Jul 31 19:15:24.250195 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:24.250151 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8" event={"ID":"a337efd3-fd30-4567-866a-72023cd150b0","Type":"ContainerDied","Data":"754c3d60042c8459e2bd3f3d39ed78a888c9e0ef72b0d8743faab00ea8550860"} Jul 31 19:15:24.250195 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:24.250174 2565 scope.go:117] "RemoveContainer" containerID="c1d3bca1afb08e190256d34860b61fe24484d416b46841ec446d482247264841" Jul 31 19:15:24.259036 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:24.259017 2565 scope.go:117] "RemoveContainer" containerID="db7733db16c83f3a4b56ab0082fb1b5a42c871e7a0b9d9e70b5caf09f581ffdf" Jul 31 19:15:24.267108 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:24.267091 2565 scope.go:117] "RemoveContainer" containerID="c1d3bca1afb08e190256d34860b61fe24484d416b46841ec446d482247264841" Jul 31 19:15:24.267468 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:15:24.267444 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1d3bca1afb08e190256d34860b61fe24484d416b46841ec446d482247264841\": container with ID starting with c1d3bca1afb08e190256d34860b61fe24484d416b46841ec446d482247264841 not found: ID does not exist" containerID="c1d3bca1afb08e190256d34860b61fe24484d416b46841ec446d482247264841" Jul 31 19:15:24.267652 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:24.267536 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1d3bca1afb08e190256d34860b61fe24484d416b46841ec446d482247264841"} err="failed to get container status \"c1d3bca1afb08e190256d34860b61fe24484d416b46841ec446d482247264841\": rpc error: code = NotFound desc = could not find container \"c1d3bca1afb08e190256d34860b61fe24484d416b46841ec446d482247264841\": container with ID starting with c1d3bca1afb08e190256d34860b61fe24484d416b46841ec446d482247264841 not found: ID does not exist" Jul 31 19:15:24.267652 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:24.267572 2565 scope.go:117] "RemoveContainer" containerID="db7733db16c83f3a4b56ab0082fb1b5a42c871e7a0b9d9e70b5caf09f581ffdf" Jul 31 19:15:24.267866 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:15:24.267847 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db7733db16c83f3a4b56ab0082fb1b5a42c871e7a0b9d9e70b5caf09f581ffdf\": container with ID starting with db7733db16c83f3a4b56ab0082fb1b5a42c871e7a0b9d9e70b5caf09f581ffdf not found: ID does not exist" containerID="db7733db16c83f3a4b56ab0082fb1b5a42c871e7a0b9d9e70b5caf09f581ffdf" Jul 31 19:15:24.267915 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:24.267873 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db7733db16c83f3a4b56ab0082fb1b5a42c871e7a0b9d9e70b5caf09f581ffdf"} err="failed to get container status \"db7733db16c83f3a4b56ab0082fb1b5a42c871e7a0b9d9e70b5caf09f581ffdf\": rpc error: code = NotFound desc = could not find container \"db7733db16c83f3a4b56ab0082fb1b5a42c871e7a0b9d9e70b5caf09f581ffdf\": container with ID starting with db7733db16c83f3a4b56ab0082fb1b5a42c871e7a0b9d9e70b5caf09f581ffdf not found: ID does not exist" Jul 31 19:15:24.269938 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:24.269906 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8"] Jul 31 19:15:24.274888 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:24.274866 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-bc42f1be-openshift-default-669944dcfb-9kvh8"] Jul 31 19:15:25.886874 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:25.886836 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a337efd3-fd30-4567-866a-72023cd150b0" path="/var/lib/kubelet/pods/a337efd3-fd30-4567-866a-72023cd150b0/volumes" Jul 31 19:15:31.050949 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:31.050913 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:15:32.752798 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:32.752760 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:15:37.774137 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:37.774096 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:15:37.872265 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:37.872207 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:15:37.986684 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:37.986652 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:15:38.049729 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:38.049639 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:15:38.487575 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:15:38.487493 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:16:04.475115 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:16:04.475082 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:16:11.215865 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:16:11.215827 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:16:11.367025 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:16:11.366984 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:16:11.478801 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:16:11.478721 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:16:27.372766 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:16:27.372728 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:16:33.664342 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:16:33.664308 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:16:33.829519 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:16:33.829483 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:16:33.932757 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:16:33.932678 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:16:44.307522 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:16:44.307413 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:16:44.407883 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:16:44.407851 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:16:44.506796 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:16:44.506763 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:16:55.456262 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:16:55.456211 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:16:57.266224 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:16:57.266194 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:16:57.336433 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:16:57.336402 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:17:01.880955 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:01.880912 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:17:01.982218 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:01.982173 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:17:02.085965 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:02.085916 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:17:02.181668 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:02.181560 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:17:02.855311 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:02.855278 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:17:02.955327 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:02.955287 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:17:03.055352 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:03.055315 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:17:04.542498 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:04.542453 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:17:05.777344 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:05.777313 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:17:11.691779 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:11.691747 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:17:11.790412 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:11.790381 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:17:12.382717 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:12.382683 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:17:13.660300 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:13.660262 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:17:19.533032 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:19.532993 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:17:19.599689 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:19.599650 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:17:19.792304 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:19.792201 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:17:20.057126 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.057050 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk"] Jul 31 19:17:20.057445 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.057428 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ece0d223-3735-4994-97c7-f6c66863b26b" containerName="cleanup" Jul 31 19:17:20.057445 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.057447 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ece0d223-3735-4994-97c7-f6c66863b26b" containerName="cleanup" Jul 31 19:17:20.057603 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.057461 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a337efd3-fd30-4567-866a-72023cd150b0" containerName="istio-proxy" Jul 31 19:17:20.057603 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.057469 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a337efd3-fd30-4567-866a-72023cd150b0" containerName="istio-proxy" Jul 31 19:17:20.057603 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.057497 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a337efd3-fd30-4567-866a-72023cd150b0" containerName="istio-proxy" Jul 31 19:17:20.057603 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.057506 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a337efd3-fd30-4567-866a-72023cd150b0" containerName="istio-proxy" Jul 31 19:17:20.057816 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.057623 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="db3c24d5-3bc4-4229-bdff-ddcaba3c12e3" containerName="istio-proxy" Jul 31 19:17:20.057816 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.057651 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a337efd3-fd30-4567-866a-72023cd150b0" containerName="istio-proxy" Jul 31 19:17:20.057816 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.057659 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a337efd3-fd30-4567-866a-72023cd150b0" containerName="istio-proxy" Jul 31 19:17:20.057816 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.057670 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ece0d223-3735-4994-97c7-f6c66863b26b" containerName="cleanup" Jul 31 19:17:20.060935 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.060916 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:20.062863 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.062846 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f249e039-openshift-default-dockercfg-kb6xx\"" Jul 31 19:17:20.083277 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.083227 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk"] Jul 31 19:17:20.086972 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.086949 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:17:20.152846 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.152810 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2799579e-30c1-44b5-ade0-ff67f37fc2cc-credential-socket\") pod \"e2e-mt-f249e039-openshift-default-77f68d4576-9nchk\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:20.153018 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.152864 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2799579e-30c1-44b5-ade0-ff67f37fc2cc-istio-token\") pod \"e2e-mt-f249e039-openshift-default-77f68d4576-9nchk\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:20.153018 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.152963 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2799579e-30c1-44b5-ade0-ff67f37fc2cc-istio-data\") pod \"e2e-mt-f249e039-openshift-default-77f68d4576-9nchk\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:20.153018 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.152997 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2799579e-30c1-44b5-ade0-ff67f37fc2cc-istio-podinfo\") pod \"e2e-mt-f249e039-openshift-default-77f68d4576-9nchk\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:20.153151 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.153057 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2799579e-30c1-44b5-ade0-ff67f37fc2cc-istio-envoy\") pod \"e2e-mt-f249e039-openshift-default-77f68d4576-9nchk\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:20.153151 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.153121 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpdkk\" (UniqueName: \"kubernetes.io/projected/2799579e-30c1-44b5-ade0-ff67f37fc2cc-kube-api-access-cpdkk\") pod \"e2e-mt-f249e039-openshift-default-77f68d4576-9nchk\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:20.153233 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.153149 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2799579e-30c1-44b5-ade0-ff67f37fc2cc-workload-certs\") pod \"e2e-mt-f249e039-openshift-default-77f68d4576-9nchk\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:20.153233 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.153172 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2799579e-30c1-44b5-ade0-ff67f37fc2cc-istiod-ca-cert\") pod \"e2e-mt-f249e039-openshift-default-77f68d4576-9nchk\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:20.153233 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.153208 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2799579e-30c1-44b5-ade0-ff67f37fc2cc-workload-socket\") pod \"e2e-mt-f249e039-openshift-default-77f68d4576-9nchk\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:20.253878 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.253847 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2799579e-30c1-44b5-ade0-ff67f37fc2cc-credential-socket\") pod \"e2e-mt-f249e039-openshift-default-77f68d4576-9nchk\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:20.254051 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.253883 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2799579e-30c1-44b5-ade0-ff67f37fc2cc-istio-token\") pod \"e2e-mt-f249e039-openshift-default-77f68d4576-9nchk\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:20.254051 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.254011 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2799579e-30c1-44b5-ade0-ff67f37fc2cc-istio-data\") pod \"e2e-mt-f249e039-openshift-default-77f68d4576-9nchk\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:20.254139 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.254058 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2799579e-30c1-44b5-ade0-ff67f37fc2cc-istio-podinfo\") pod \"e2e-mt-f249e039-openshift-default-77f68d4576-9nchk\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:20.254139 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.254093 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2799579e-30c1-44b5-ade0-ff67f37fc2cc-istio-envoy\") pod \"e2e-mt-f249e039-openshift-default-77f68d4576-9nchk\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:20.254139 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.254123 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpdkk\" (UniqueName: \"kubernetes.io/projected/2799579e-30c1-44b5-ade0-ff67f37fc2cc-kube-api-access-cpdkk\") pod \"e2e-mt-f249e039-openshift-default-77f68d4576-9nchk\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:20.254327 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.254154 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2799579e-30c1-44b5-ade0-ff67f37fc2cc-workload-certs\") pod \"e2e-mt-f249e039-openshift-default-77f68d4576-9nchk\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:20.254327 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.254178 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2799579e-30c1-44b5-ade0-ff67f37fc2cc-istiod-ca-cert\") pod \"e2e-mt-f249e039-openshift-default-77f68d4576-9nchk\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:20.254327 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.254219 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2799579e-30c1-44b5-ade0-ff67f37fc2cc-workload-socket\") pod \"e2e-mt-f249e039-openshift-default-77f68d4576-9nchk\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:20.254327 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.254287 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2799579e-30c1-44b5-ade0-ff67f37fc2cc-credential-socket\") pod \"e2e-mt-f249e039-openshift-default-77f68d4576-9nchk\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:20.254533 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.254404 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2799579e-30c1-44b5-ade0-ff67f37fc2cc-istio-data\") pod \"e2e-mt-f249e039-openshift-default-77f68d4576-9nchk\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:20.254533 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.254516 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2799579e-30c1-44b5-ade0-ff67f37fc2cc-workload-certs\") pod \"e2e-mt-f249e039-openshift-default-77f68d4576-9nchk\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:20.254640 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.254588 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2799579e-30c1-44b5-ade0-ff67f37fc2cc-workload-socket\") pod \"e2e-mt-f249e039-openshift-default-77f68d4576-9nchk\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:20.254877 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.254861 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2799579e-30c1-44b5-ade0-ff67f37fc2cc-istiod-ca-cert\") pod \"e2e-mt-f249e039-openshift-default-77f68d4576-9nchk\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:20.256954 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.256925 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2799579e-30c1-44b5-ade0-ff67f37fc2cc-istio-envoy\") pod \"e2e-mt-f249e039-openshift-default-77f68d4576-9nchk\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:20.257197 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.257176 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2799579e-30c1-44b5-ade0-ff67f37fc2cc-istio-podinfo\") pod \"e2e-mt-f249e039-openshift-default-77f68d4576-9nchk\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:20.262395 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.262374 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2799579e-30c1-44b5-ade0-ff67f37fc2cc-istio-token\") pod \"e2e-mt-f249e039-openshift-default-77f68d4576-9nchk\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:20.262503 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.262438 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpdkk\" (UniqueName: \"kubernetes.io/projected/2799579e-30c1-44b5-ade0-ff67f37fc2cc-kube-api-access-cpdkk\") pod \"e2e-mt-f249e039-openshift-default-77f68d4576-9nchk\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:20.296922 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.296886 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:17:20.370937 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.370856 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:20.518057 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.518033 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk"] Jul 31 19:17:20.520140 ip-10-0-139-39 kubenswrapper[2565]: W0731 19:17:20.520109 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2799579e_30c1_44b5_ade0_ff67f37fc2cc.slice/crio-8e3bd6049c840ce23bd3cba85b56ad1279e6ed57df3e4f5e214649f1c464ee72 WatchSource:0}: Error finding container 8e3bd6049c840ce23bd3cba85b56ad1279e6ed57df3e4f5e214649f1c464ee72: Status 404 returned error can't find the container with id 8e3bd6049c840ce23bd3cba85b56ad1279e6ed57df3e4f5e214649f1c464ee72 Jul 31 19:17:20.522329 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.522298 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:17:20.522430 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.522361 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:17:20.522430 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.522389 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:17:20.837542 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.837503 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" event={"ID":"2799579e-30c1-44b5-ade0-ff67f37fc2cc","Type":"ContainerStarted","Data":"52dfd139df91ccb628cfa1293a2373474dbdc572816c815b6f8dba7e2512322e"} Jul 31 19:17:20.837926 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.837560 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" event={"ID":"2799579e-30c1-44b5-ade0-ff67f37fc2cc","Type":"ContainerStarted","Data":"8e3bd6049c840ce23bd3cba85b56ad1279e6ed57df3e4f5e214649f1c464ee72"} Jul 31 19:17:20.857041 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:20.856981 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" podStartSLOduration=0.856966491 podStartE2EDuration="856.966491ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:17:20.855986871 +0000 UTC m=+2017.493976674" watchObservedRunningTime="2026-07-31 19:17:20.856966491 +0000 UTC m=+2017.494956269" Jul 31 19:17:21.371739 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:21.371705 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:21.376712 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:21.376685 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:21.843785 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:21.843746 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:21.844680 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:21.844653 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:22.783293 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:22.783222 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:17:22.891098 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:22.891067 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:17:23.910165 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:23.910135 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:17:24.089563 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:24.089524 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:17:24.613824 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:24.613789 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:17:24.708043 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:24.708007 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:17:27.483249 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:27.483213 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:17:37.008860 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:37.007761 2565 patch_prober.go:28] interesting pod/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": read tcp 10.133.0.2:53662->10.133.0.54:15021: read: connection reset by peer" start-of-body= Jul 31 19:17:37.008860 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:37.007826 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" podUID="2799579e-30c1-44b5-ade0-ff67f37fc2cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": read tcp 10.133.0.2:53662->10.133.0.54:15021: read: connection reset by peer" Jul 31 19:17:37.924227 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:37.924192 2565 generic.go:356] "Generic (PLEG): container finished" podID="2799579e-30c1-44b5-ade0-ff67f37fc2cc" containerID="52dfd139df91ccb628cfa1293a2373474dbdc572816c815b6f8dba7e2512322e" exitCode=0 Jul 31 19:17:37.924419 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:37.924265 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" event={"ID":"2799579e-30c1-44b5-ade0-ff67f37fc2cc","Type":"ContainerDied","Data":"52dfd139df91ccb628cfa1293a2373474dbdc572816c815b6f8dba7e2512322e"} Jul 31 19:17:37.924693 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:37.924674 2565 scope.go:117] "RemoveContainer" containerID="52dfd139df91ccb628cfa1293a2373474dbdc572816c815b6f8dba7e2512322e" Jul 31 19:17:37.925897 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:37.925713 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:17:37.925897 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:37.925799 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:17:37.925897 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:37.925851 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:17:38.933654 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:38.933613 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" event={"ID":"2799579e-30c1-44b5-ade0-ff67f37fc2cc","Type":"ContainerStarted","Data":"76327b36e29567a2042326b627a96b6e6717c325a857e007ba463e794d22d0fb"} Jul 31 19:17:39.371647 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:39.371611 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:39.371868 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:39.371845 2565 patch_prober.go:28] interesting pod/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 31 19:17:39.371919 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:39.371899 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" podUID="2799579e-30c1-44b5-ade0-ff67f37fc2cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 31 19:17:39.497399 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:39.497369 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:17:40.371134 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:40.371101 2565 patch_prober.go:28] interesting pod/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 31 19:17:40.371540 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:40.371159 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" podUID="2799579e-30c1-44b5-ade0-ff67f37fc2cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 31 19:17:41.371948 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:41.371906 2565 patch_prober.go:28] interesting pod/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 31 19:17:41.372367 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:41.371971 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" podUID="2799579e-30c1-44b5-ade0-ff67f37fc2cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 31 19:17:42.371260 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:42.371209 2565 patch_prober.go:28] interesting pod/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 31 19:17:42.371449 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:42.371294 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" podUID="2799579e-30c1-44b5-ade0-ff67f37fc2cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 31 19:17:43.372162 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:43.372126 2565 patch_prober.go:28] interesting pod/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 31 19:17:43.372568 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:43.372195 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" podUID="2799579e-30c1-44b5-ade0-ff67f37fc2cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 31 19:17:44.374756 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:44.374726 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:44.375195 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:44.375119 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:44.375873 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:44.375851 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:17:47.742587 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:47.742550 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:17:48.392121 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:48.392060 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:17:48.490642 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:48.490593 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:17:48.844912 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:48.844875 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:17:55.090849 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:55.090797 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:17:55.151859 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:17:55.151817 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:18:06.295412 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:06.295328 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:18:06.366505 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:06.366468 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:18:06.430423 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:06.430383 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:18:06.493294 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:06.493230 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:18:06.844433 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:06.844378 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:18:08.265935 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:08.265899 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:18:13.133901 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:13.133861 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:18:13.205166 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:13.205126 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:18:13.361642 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:13.361610 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk"] Jul 31 19:18:13.361854 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:13.361831 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" podUID="2799579e-30c1-44b5-ade0-ff67f37fc2cc" containerName="istio-proxy" containerID="cri-o://76327b36e29567a2042326b627a96b6e6717c325a857e007ba463e794d22d0fb" gracePeriod=30 Jul 31 19:18:13.387376 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:13.387303 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:18:13.589247 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:13.589215 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:18:13.709823 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:13.709737 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:18:13.809621 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:13.809586 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:18:13.936019 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:13.935966 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:18:14.377186 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:14.377152 2565 patch_prober.go:28] interesting pod/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 31 19:18:14.377641 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:14.377218 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" podUID="2799579e-30c1-44b5-ade0-ff67f37fc2cc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 31 19:18:18.627765 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:18.627704 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:18:18.809097 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:18.809060 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2799579e-30c1-44b5-ade0-ff67f37fc2cc-istiod-ca-cert\") pod \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " Jul 31 19:18:18.809097 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:18.809097 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2799579e-30c1-44b5-ade0-ff67f37fc2cc-workload-socket\") pod \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " Jul 31 19:18:18.809380 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:18.809117 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2799579e-30c1-44b5-ade0-ff67f37fc2cc-istio-data\") pod \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " Jul 31 19:18:18.809380 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:18.809169 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2799579e-30c1-44b5-ade0-ff67f37fc2cc-workload-certs\") pod \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " Jul 31 19:18:18.809380 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:18.809194 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2799579e-30c1-44b5-ade0-ff67f37fc2cc-istio-envoy\") pod \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " Jul 31 19:18:18.809380 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:18.809232 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpdkk\" (UniqueName: \"kubernetes.io/projected/2799579e-30c1-44b5-ade0-ff67f37fc2cc-kube-api-access-cpdkk\") pod \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " Jul 31 19:18:18.809380 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:18.809356 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2799579e-30c1-44b5-ade0-ff67f37fc2cc-istio-token\") pod \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " Jul 31 19:18:18.809380 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:18.809380 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2799579e-30c1-44b5-ade0-ff67f37fc2cc-istio-podinfo\") pod \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " Jul 31 19:18:18.809707 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:18.809411 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2799579e-30c1-44b5-ade0-ff67f37fc2cc-credential-socket\") pod \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\" (UID: \"2799579e-30c1-44b5-ade0-ff67f37fc2cc\") " Jul 31 19:18:18.809707 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:18.809507 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2799579e-30c1-44b5-ade0-ff67f37fc2cc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2799579e-30c1-44b5-ade0-ff67f37fc2cc" (UID: "2799579e-30c1-44b5-ade0-ff67f37fc2cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:18:18.809707 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:18.809641 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2799579e-30c1-44b5-ade0-ff67f37fc2cc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2799579e-30c1-44b5-ade0-ff67f37fc2cc" (UID: "2799579e-30c1-44b5-ade0-ff67f37fc2cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 19:18:18.809862 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:18.809651 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2799579e-30c1-44b5-ade0-ff67f37fc2cc-istio-data" (OuterVolumeSpecName: "istio-data") pod "2799579e-30c1-44b5-ade0-ff67f37fc2cc" (UID: "2799579e-30c1-44b5-ade0-ff67f37fc2cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:18:18.809862 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:18.809735 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2799579e-30c1-44b5-ade0-ff67f37fc2cc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2799579e-30c1-44b5-ade0-ff67f37fc2cc" (UID: "2799579e-30c1-44b5-ade0-ff67f37fc2cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:18:18.809862 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:18.809786 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2799579e-30c1-44b5-ade0-ff67f37fc2cc-istiod-ca-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:18:18.809862 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:18.809808 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2799579e-30c1-44b5-ade0-ff67f37fc2cc-workload-socket\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:18:18.809862 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:18.809823 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2799579e-30c1-44b5-ade0-ff67f37fc2cc-istio-data\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:18:18.810046 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:18.809895 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2799579e-30c1-44b5-ade0-ff67f37fc2cc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2799579e-30c1-44b5-ade0-ff67f37fc2cc" (UID: "2799579e-30c1-44b5-ade0-ff67f37fc2cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:18:18.811595 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:18.811569 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2799579e-30c1-44b5-ade0-ff67f37fc2cc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2799579e-30c1-44b5-ade0-ff67f37fc2cc" (UID: "2799579e-30c1-44b5-ade0-ff67f37fc2cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:18:18.811685 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:18.811573 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2799579e-30c1-44b5-ade0-ff67f37fc2cc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2799579e-30c1-44b5-ade0-ff67f37fc2cc" (UID: "2799579e-30c1-44b5-ade0-ff67f37fc2cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 19:18:18.811887 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:18.811870 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2799579e-30c1-44b5-ade0-ff67f37fc2cc-kube-api-access-cpdkk" (OuterVolumeSpecName: "kube-api-access-cpdkk") pod "2799579e-30c1-44b5-ade0-ff67f37fc2cc" (UID: "2799579e-30c1-44b5-ade0-ff67f37fc2cc"). InnerVolumeSpecName "kube-api-access-cpdkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:18:18.811935 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:18.811897 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2799579e-30c1-44b5-ade0-ff67f37fc2cc-istio-token" (OuterVolumeSpecName: "istio-token") pod "2799579e-30c1-44b5-ade0-ff67f37fc2cc" (UID: "2799579e-30c1-44b5-ade0-ff67f37fc2cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:18:18.910725 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:18.910628 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2799579e-30c1-44b5-ade0-ff67f37fc2cc-istio-token\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:18:18.910725 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:18.910669 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2799579e-30c1-44b5-ade0-ff67f37fc2cc-istio-podinfo\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:18:18.910725 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:18.910684 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2799579e-30c1-44b5-ade0-ff67f37fc2cc-credential-socket\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:18:18.910725 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:18.910699 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2799579e-30c1-44b5-ade0-ff67f37fc2cc-workload-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:18:18.910725 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:18.910712 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2799579e-30c1-44b5-ade0-ff67f37fc2cc-istio-envoy\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:18:18.910725 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:18.910721 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpdkk\" (UniqueName: \"kubernetes.io/projected/2799579e-30c1-44b5-ade0-ff67f37fc2cc-kube-api-access-cpdkk\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:18:19.143251 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:19.143214 2565 generic.go:356] "Generic (PLEG): container finished" podID="2799579e-30c1-44b5-ade0-ff67f37fc2cc" containerID="76327b36e29567a2042326b627a96b6e6717c325a857e007ba463e794d22d0fb" exitCode=0 Jul 31 19:18:19.143424 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:19.143283 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" event={"ID":"2799579e-30c1-44b5-ade0-ff67f37fc2cc","Type":"ContainerDied","Data":"76327b36e29567a2042326b627a96b6e6717c325a857e007ba463e794d22d0fb"} Jul 31 19:18:19.143424 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:19.143293 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" Jul 31 19:18:19.143424 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:19.143345 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk" event={"ID":"2799579e-30c1-44b5-ade0-ff67f37fc2cc","Type":"ContainerDied","Data":"8e3bd6049c840ce23bd3cba85b56ad1279e6ed57df3e4f5e214649f1c464ee72"} Jul 31 19:18:19.143424 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:19.143367 2565 scope.go:117] "RemoveContainer" containerID="76327b36e29567a2042326b627a96b6e6717c325a857e007ba463e794d22d0fb" Jul 31 19:18:19.153097 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:19.153078 2565 scope.go:117] "RemoveContainer" containerID="52dfd139df91ccb628cfa1293a2373474dbdc572816c815b6f8dba7e2512322e" Jul 31 19:18:19.160864 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:19.160845 2565 scope.go:117] "RemoveContainer" containerID="76327b36e29567a2042326b627a96b6e6717c325a857e007ba463e794d22d0fb" Jul 31 19:18:19.161137 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:18:19.161119 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76327b36e29567a2042326b627a96b6e6717c325a857e007ba463e794d22d0fb\": container with ID starting with 76327b36e29567a2042326b627a96b6e6717c325a857e007ba463e794d22d0fb not found: ID does not exist" containerID="76327b36e29567a2042326b627a96b6e6717c325a857e007ba463e794d22d0fb" Jul 31 19:18:19.161182 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:19.161145 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76327b36e29567a2042326b627a96b6e6717c325a857e007ba463e794d22d0fb"} err="failed to get container status \"76327b36e29567a2042326b627a96b6e6717c325a857e007ba463e794d22d0fb\": rpc error: code = NotFound desc = could not find container \"76327b36e29567a2042326b627a96b6e6717c325a857e007ba463e794d22d0fb\": container with ID starting with 76327b36e29567a2042326b627a96b6e6717c325a857e007ba463e794d22d0fb not found: ID does not exist" Jul 31 19:18:19.161182 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:19.161165 2565 scope.go:117] "RemoveContainer" containerID="52dfd139df91ccb628cfa1293a2373474dbdc572816c815b6f8dba7e2512322e" Jul 31 19:18:19.161425 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:18:19.161407 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52dfd139df91ccb628cfa1293a2373474dbdc572816c815b6f8dba7e2512322e\": container with ID starting with 52dfd139df91ccb628cfa1293a2373474dbdc572816c815b6f8dba7e2512322e not found: ID does not exist" containerID="52dfd139df91ccb628cfa1293a2373474dbdc572816c815b6f8dba7e2512322e" Jul 31 19:18:19.161481 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:19.161430 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52dfd139df91ccb628cfa1293a2373474dbdc572816c815b6f8dba7e2512322e"} err="failed to get container status \"52dfd139df91ccb628cfa1293a2373474dbdc572816c815b6f8dba7e2512322e\": rpc error: code = NotFound desc = could not find container \"52dfd139df91ccb628cfa1293a2373474dbdc572816c815b6f8dba7e2512322e\": container with ID starting with 52dfd139df91ccb628cfa1293a2373474dbdc572816c815b6f8dba7e2512322e not found: ID does not exist" Jul 31 19:18:19.166838 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:19.166819 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk"] Jul 31 19:18:19.173081 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:19.173061 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f249e039-openshift-default-77f68d4576-9nchk"] Jul 31 19:18:19.887692 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:19.887658 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2799579e-30c1-44b5-ade0-ff67f37fc2cc" path="/var/lib/kubelet/pods/2799579e-30c1-44b5-ade0-ff67f37fc2cc/volumes" Jul 31 19:18:20.030212 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:20.030175 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:18:20.089360 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:20.089327 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:18:20.180921 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:20.180836 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:18:20.279140 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:20.279110 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:18:20.387002 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:20.386973 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:18:26.497948 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:26.497910 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:18:28.764812 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:28.764778 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:18:45.659418 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:18:45.659381 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:20:29.228474 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.228435 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:20:29.456733 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.456702 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv"] Jul 31 19:20:29.457074 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.457063 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2799579e-30c1-44b5-ade0-ff67f37fc2cc" containerName="istio-proxy" Jul 31 19:20:29.457118 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.457076 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2799579e-30c1-44b5-ade0-ff67f37fc2cc" containerName="istio-proxy" Jul 31 19:20:29.457172 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.457163 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2799579e-30c1-44b5-ade0-ff67f37fc2cc" containerName="istio-proxy" Jul 31 19:20:29.460198 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.460182 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:29.462116 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.462095 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-aaeacbf1-openshift-default-dockercfg-hn76l\"" Jul 31 19:20:29.471315 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.471292 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv"] Jul 31 19:20:29.480875 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.480815 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:20:29.558223 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.558187 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:20:29.590094 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.590057 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-istiod-ca-cert\") pod \"e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:29.590312 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.590115 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-workload-socket\") pod \"e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:29.590312 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.590169 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-istio-podinfo\") pod \"e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:29.590312 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.590223 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-istio-token\") pod \"e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:29.590312 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.590308 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-credential-socket\") pod \"e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:29.590491 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.590335 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-workload-certs\") pod \"e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:29.590491 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.590368 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-istio-envoy\") pod \"e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:29.590491 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.590401 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-istio-data\") pod \"e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:29.590491 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.590421 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsp78\" (UniqueName: \"kubernetes.io/projected/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-kube-api-access-gsp78\") pod \"e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:29.684253 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.684217 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:20:29.691515 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.691471 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-istiod-ca-cert\") pod \"e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:29.691709 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.691559 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-workload-socket\") pod \"e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:29.691709 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.691585 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-istio-podinfo\") pod \"e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:29.691709 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.691609 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-istio-token\") pod \"e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:29.691894 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.691810 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-credential-socket\") pod \"e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:29.691894 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.691858 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-workload-certs\") pod \"e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:29.691894 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.691889 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-istio-envoy\") pod \"e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:29.692056 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.691919 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-istio-data\") pod \"e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:29.692056 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.691948 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsp78\" (UniqueName: \"kubernetes.io/projected/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-kube-api-access-gsp78\") pod \"e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:29.692216 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.692052 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-workload-socket\") pod \"e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:29.692216 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.692156 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-credential-socket\") pod \"e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:29.692380 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.692334 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-workload-certs\") pod \"e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:29.692436 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.692419 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-istio-data\") pod \"e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:29.693104 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.693081 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-istiod-ca-cert\") pod \"e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:29.694933 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.694903 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-istio-envoy\") pod \"e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:29.695267 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.695231 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-istio-podinfo\") pod \"e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:29.705226 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.705201 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-istio-token\") pod \"e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:29.705808 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.705782 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsp78\" (UniqueName: \"kubernetes.io/projected/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-kube-api-access-gsp78\") pod \"e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:29.769823 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.769789 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:29.920128 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.920098 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv"] Jul 31 19:20:29.922182 ip-10-0-139-39 kubenswrapper[2565]: W0731 19:20:29.922154 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda2bf329_b9d0_4c78_8e58_afa8bdef24bb.slice/crio-9059a98f4b528374c9340ff1ab2de5a0f9d497bba0b44557e2084075cfa4de76 WatchSource:0}: Error finding container 9059a98f4b528374c9340ff1ab2de5a0f9d497bba0b44557e2084075cfa4de76: Status 404 returned error can't find the container with id 9059a98f4b528374c9340ff1ab2de5a0f9d497bba0b44557e2084075cfa4de76 Jul 31 19:20:29.927286 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.924761 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:20:29.927286 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.924915 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:20:29.927286 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:29.925001 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:20:30.790762 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:30.790720 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" event={"ID":"da2bf329-b9d0-4c78-8e58-afa8bdef24bb","Type":"ContainerStarted","Data":"18f7330001e02f2918193f71fd78803e17cc6de7e63bc2dc280208d84e648514"} Jul 31 19:20:30.791172 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:30.790820 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" event={"ID":"da2bf329-b9d0-4c78-8e58-afa8bdef24bb","Type":"ContainerStarted","Data":"9059a98f4b528374c9340ff1ab2de5a0f9d497bba0b44557e2084075cfa4de76"} Jul 31 19:20:30.814141 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:30.814093 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" podStartSLOduration=1.8140797690000001 podStartE2EDuration="1.814079769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:20:30.811814043 +0000 UTC m=+2207.449803844" watchObservedRunningTime="2026-07-31 19:20:30.814079769 +0000 UTC m=+2207.452069570" Jul 31 19:20:31.770122 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:31.770088 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:31.775387 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:31.775360 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:31.796380 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:31.796352 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:31.797576 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:31.797549 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:32.824596 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:32.824565 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:20:32.960757 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:32.960724 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:20:33.888005 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:33.887968 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:20:34.080397 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:34.080362 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:20:34.431180 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:34.431132 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:20:34.532449 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:34.532416 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:20:37.058455 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:37.058414 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:20:44.059815 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:44.059781 2565 scope.go:117] "RemoveContainer" containerID="aed4dbc8d371515e0681669b261f48df4b778284ee6cd0e60f77bdbeb4dc6f60" Jul 31 19:20:46.798117 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:46.798041 2565 patch_prober.go:28] interesting pod/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 31 19:20:46.798117 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:46.798100 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" podUID="da2bf329-b9d0-4c78-8e58-afa8bdef24bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 31 19:20:46.886377 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:46.886286 2565 generic.go:356] "Generic (PLEG): container finished" podID="da2bf329-b9d0-4c78-8e58-afa8bdef24bb" containerID="18f7330001e02f2918193f71fd78803e17cc6de7e63bc2dc280208d84e648514" exitCode=0 Jul 31 19:20:46.886565 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:46.886364 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" event={"ID":"da2bf329-b9d0-4c78-8e58-afa8bdef24bb","Type":"ContainerDied","Data":"18f7330001e02f2918193f71fd78803e17cc6de7e63bc2dc280208d84e648514"} Jul 31 19:20:46.886779 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:46.886765 2565 scope.go:117] "RemoveContainer" containerID="18f7330001e02f2918193f71fd78803e17cc6de7e63bc2dc280208d84e648514" Jul 31 19:20:46.887900 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:46.887872 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:20:46.887975 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:46.887956 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:20:46.888014 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:46.888001 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:20:47.892392 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:47.892351 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" event={"ID":"da2bf329-b9d0-4c78-8e58-afa8bdef24bb","Type":"ContainerStarted","Data":"6d46d9141aad9c1dc11b4ad04b272e645f1dfef342d63c8fc1252011b9972ac7"} Jul 31 19:20:48.770258 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:48.770190 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:48.770484 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:48.770460 2565 patch_prober.go:28] interesting pod/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 31 19:20:48.770532 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:48.770513 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" podUID="da2bf329-b9d0-4c78-8e58-afa8bdef24bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 31 19:20:49.770479 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:49.770440 2565 patch_prober.go:28] interesting pod/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 31 19:20:49.770889 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:49.770514 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" podUID="da2bf329-b9d0-4c78-8e58-afa8bdef24bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 31 19:20:50.771183 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:50.771142 2565 patch_prober.go:28] interesting pod/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 31 19:20:50.771607 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:50.771219 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" podUID="da2bf329-b9d0-4c78-8e58-afa8bdef24bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 31 19:20:51.770430 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:51.770392 2565 patch_prober.go:28] interesting pod/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 31 19:20:51.770612 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:51.770448 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" podUID="da2bf329-b9d0-4c78-8e58-afa8bdef24bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 31 19:20:52.771131 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:52.771091 2565 patch_prober.go:28] interesting pod/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 31 19:20:52.771543 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:52.771150 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" podUID="da2bf329-b9d0-4c78-8e58-afa8bdef24bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 31 19:20:53.774137 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:53.774106 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:53.774564 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:53.774456 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:53.774972 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:53.774952 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:20:57.677253 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:20:57.677213 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:21:02.820034 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:02.819996 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh"] Jul 31 19:21:02.820441 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:02.820383 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2799579e-30c1-44b5-ade0-ff67f37fc2cc" containerName="istio-proxy" Jul 31 19:21:02.820441 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:02.820395 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2799579e-30c1-44b5-ade0-ff67f37fc2cc" containerName="istio-proxy" Jul 31 19:21:02.820515 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:02.820466 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2799579e-30c1-44b5-ade0-ff67f37fc2cc" containerName="istio-proxy" Jul 31 19:21:02.822649 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:02.822634 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh" Jul 31 19:21:02.825419 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:02.825390 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-aaeacbf1\"/\"default-dockercfg-qnjxf\"" Jul 31 19:21:02.825419 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:02.825400 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-aaeacbf1\"/\"e2e-lifecycle-model-aaeacbf1-kserve-self-signed-certs\"" Jul 31 19:21:02.825633 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:02.825399 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-aaeacbf1\"/\"openshift-service-ca.crt\"" Jul 31 19:21:02.825633 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:02.825441 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-aaeacbf1\"/\"kube-root-ca.crt\"" Jul 31 19:21:02.837168 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:02.837146 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh"] Jul 31 19:21:02.850131 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:02.850100 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:21:02.905251 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:02.905214 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drv76\" (UniqueName: \"kubernetes.io/projected/a53b0913-db93-4105-ba34-5cce98fc8cc5-kube-api-access-drv76\") pod \"e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh\" (UID: \"a53b0913-db93-4105-ba34-5cce98fc8cc5\") " pod="ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh" Jul 31 19:21:02.905414 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:02.905274 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a53b0913-db93-4105-ba34-5cce98fc8cc5-model-cache\") pod \"e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh\" (UID: \"a53b0913-db93-4105-ba34-5cce98fc8cc5\") " pod="ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh" Jul 31 19:21:02.905414 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:02.905325 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a53b0913-db93-4105-ba34-5cce98fc8cc5-dshm\") pod \"e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh\" (UID: \"a53b0913-db93-4105-ba34-5cce98fc8cc5\") " pod="ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh" Jul 31 19:21:02.905414 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:02.905357 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53b0913-db93-4105-ba34-5cce98fc8cc5-tls-certs\") pod \"e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh\" (UID: \"a53b0913-db93-4105-ba34-5cce98fc8cc5\") " pod="ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh" Jul 31 19:21:02.905414 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:02.905402 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a53b0913-db93-4105-ba34-5cce98fc8cc5-tmp-dir\") pod \"e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh\" (UID: \"a53b0913-db93-4105-ba34-5cce98fc8cc5\") " pod="ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh" Jul 31 19:21:02.905564 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:02.905429 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a53b0913-db93-4105-ba34-5cce98fc8cc5-home\") pod \"e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh\" (UID: \"a53b0913-db93-4105-ba34-5cce98fc8cc5\") " pod="ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh" Jul 31 19:21:03.006307 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:03.006266 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a53b0913-db93-4105-ba34-5cce98fc8cc5-tmp-dir\") pod \"e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh\" (UID: \"a53b0913-db93-4105-ba34-5cce98fc8cc5\") " pod="ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh" Jul 31 19:21:03.006498 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:03.006363 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a53b0913-db93-4105-ba34-5cce98fc8cc5-home\") pod \"e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh\" (UID: \"a53b0913-db93-4105-ba34-5cce98fc8cc5\") " pod="ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh" Jul 31 19:21:03.006573 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:03.006510 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drv76\" (UniqueName: \"kubernetes.io/projected/a53b0913-db93-4105-ba34-5cce98fc8cc5-kube-api-access-drv76\") pod \"e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh\" (UID: \"a53b0913-db93-4105-ba34-5cce98fc8cc5\") " pod="ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh" Jul 31 19:21:03.006633 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:03.006587 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a53b0913-db93-4105-ba34-5cce98fc8cc5-model-cache\") pod \"e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh\" (UID: \"a53b0913-db93-4105-ba34-5cce98fc8cc5\") " pod="ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh" Jul 31 19:21:03.006633 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:03.006621 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a53b0913-db93-4105-ba34-5cce98fc8cc5-dshm\") pod \"e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh\" (UID: \"a53b0913-db93-4105-ba34-5cce98fc8cc5\") " pod="ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh" Jul 31 19:21:03.006741 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:03.006647 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53b0913-db93-4105-ba34-5cce98fc8cc5-tls-certs\") pod \"e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh\" (UID: \"a53b0913-db93-4105-ba34-5cce98fc8cc5\") " pod="ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh" Jul 31 19:21:03.006822 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:03.006782 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a53b0913-db93-4105-ba34-5cce98fc8cc5-tmp-dir\") pod \"e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh\" (UID: \"a53b0913-db93-4105-ba34-5cce98fc8cc5\") " pod="ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh" Jul 31 19:21:03.006888 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:03.006836 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a53b0913-db93-4105-ba34-5cce98fc8cc5-home\") pod \"e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh\" (UID: \"a53b0913-db93-4105-ba34-5cce98fc8cc5\") " pod="ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh" Jul 31 19:21:03.007231 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:03.007206 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a53b0913-db93-4105-ba34-5cce98fc8cc5-model-cache\") pod \"e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh\" (UID: \"a53b0913-db93-4105-ba34-5cce98fc8cc5\") " pod="ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh" Jul 31 19:21:03.008937 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:03.008919 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a53b0913-db93-4105-ba34-5cce98fc8cc5-dshm\") pod \"e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh\" (UID: \"a53b0913-db93-4105-ba34-5cce98fc8cc5\") " pod="ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh" Jul 31 19:21:03.009067 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:03.009049 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53b0913-db93-4105-ba34-5cce98fc8cc5-tls-certs\") pod \"e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh\" (UID: \"a53b0913-db93-4105-ba34-5cce98fc8cc5\") " pod="ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh" Jul 31 19:21:03.014974 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:03.014954 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drv76\" (UniqueName: \"kubernetes.io/projected/a53b0913-db93-4105-ba34-5cce98fc8cc5-kube-api-access-drv76\") pod \"e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh\" (UID: \"a53b0913-db93-4105-ba34-5cce98fc8cc5\") " pod="ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh" Jul 31 19:21:03.132526 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:03.132426 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh" Jul 31 19:21:03.270386 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:03.270356 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh"] Jul 31 19:21:03.272188 ip-10-0-139-39 kubenswrapper[2565]: W0731 19:21:03.272162 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda53b0913_db93_4105_ba34_5cce98fc8cc5.slice/crio-549ed42fc22d06aba68aeb51291938266baab29178fbe279354edc3afd4cb2bc WatchSource:0}: Error finding container 549ed42fc22d06aba68aeb51291938266baab29178fbe279354edc3afd4cb2bc: Status 404 returned error can't find the container with id 549ed42fc22d06aba68aeb51291938266baab29178fbe279354edc3afd4cb2bc Jul 31 19:21:03.975113 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:03.975074 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh" event={"ID":"a53b0913-db93-4105-ba34-5cce98fc8cc5","Type":"ContainerStarted","Data":"1e54df18eb0ae1df4381c3cf28a236360660ae199849c081867fa97344dd517f"} Jul 31 19:21:03.975113 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:03.975116 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh" event={"ID":"a53b0913-db93-4105-ba34-5cce98fc8cc5","Type":"ContainerStarted","Data":"549ed42fc22d06aba68aeb51291938266baab29178fbe279354edc3afd4cb2bc"} Jul 31 19:21:03.995074 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:03.995023 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh" podStartSLOduration=1.99500783 podStartE2EDuration="1.99500783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:21:03.993613356 +0000 UTC m=+2240.631603156" watchObservedRunningTime="2026-07-31 19:21:03.99500783 +0000 UTC m=+2240.632997629" Jul 31 19:21:05.056292 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:05.056256 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:21:13.133685 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:13.133597 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh" Jul 31 19:21:13.133685 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:13.133659 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh" Jul 31 19:21:13.146522 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:13.146490 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh" Jul 31 19:21:14.034809 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:14.034780 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh" Jul 31 19:21:21.352269 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:21.352217 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:21:44.081468 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:44.081437 2565 scope.go:117] "RemoveContainer" containerID="55655d51572cf8bfef4006242bfc0b5733d98137026e62ad9e0060bae0ec2ec5" Jul 31 19:21:44.960291 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:44.960254 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:21:47.527630 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:47.527597 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:21:47.674066 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:47.674027 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:21:47.770149 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:47.770112 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:21:47.835434 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:47.835352 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:21:50.952211 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:50.952171 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:21:52.456212 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:52.456176 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:21:55.959597 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:55.959565 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh"] Jul 31 19:21:56.040792 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:21:56.040761 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:22:10.978421 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:10.978378 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh" podUID="a53b0913-db93-4105-ba34-5cce98fc8cc5" containerName="main" containerID="cri-o://1e54df18eb0ae1df4381c3cf28a236360660ae199849c081867fa97344dd517f" gracePeriod=45 Jul 31 19:22:11.231647 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:11.231586 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh" Jul 31 19:22:11.259279 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:11.259230 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53b0913-db93-4105-ba34-5cce98fc8cc5-tls-certs\") pod \"a53b0913-db93-4105-ba34-5cce98fc8cc5\" (UID: \"a53b0913-db93-4105-ba34-5cce98fc8cc5\") " Jul 31 19:22:11.259433 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:11.259326 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a53b0913-db93-4105-ba34-5cce98fc8cc5-dshm\") pod \"a53b0913-db93-4105-ba34-5cce98fc8cc5\" (UID: \"a53b0913-db93-4105-ba34-5cce98fc8cc5\") " Jul 31 19:22:11.259433 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:11.259357 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a53b0913-db93-4105-ba34-5cce98fc8cc5-tmp-dir\") pod \"a53b0913-db93-4105-ba34-5cce98fc8cc5\" (UID: \"a53b0913-db93-4105-ba34-5cce98fc8cc5\") " Jul 31 19:22:11.259550 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:11.259447 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a53b0913-db93-4105-ba34-5cce98fc8cc5-home\") pod \"a53b0913-db93-4105-ba34-5cce98fc8cc5\" (UID: \"a53b0913-db93-4105-ba34-5cce98fc8cc5\") " Jul 31 19:22:11.259550 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:11.259520 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drv76\" (UniqueName: \"kubernetes.io/projected/a53b0913-db93-4105-ba34-5cce98fc8cc5-kube-api-access-drv76\") pod \"a53b0913-db93-4105-ba34-5cce98fc8cc5\" (UID: \"a53b0913-db93-4105-ba34-5cce98fc8cc5\") " Jul 31 19:22:11.259659 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:11.259547 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a53b0913-db93-4105-ba34-5cce98fc8cc5-model-cache\") pod \"a53b0913-db93-4105-ba34-5cce98fc8cc5\" (UID: \"a53b0913-db93-4105-ba34-5cce98fc8cc5\") " Jul 31 19:22:11.259711 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:11.259664 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53b0913-db93-4105-ba34-5cce98fc8cc5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a53b0913-db93-4105-ba34-5cce98fc8cc5" (UID: "a53b0913-db93-4105-ba34-5cce98fc8cc5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:22:11.259838 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:11.259813 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53b0913-db93-4105-ba34-5cce98fc8cc5-home" (OuterVolumeSpecName: "home") pod "a53b0913-db93-4105-ba34-5cce98fc8cc5" (UID: "a53b0913-db93-4105-ba34-5cce98fc8cc5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:22:11.259929 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:11.259871 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53b0913-db93-4105-ba34-5cce98fc8cc5-model-cache" (OuterVolumeSpecName: "model-cache") pod "a53b0913-db93-4105-ba34-5cce98fc8cc5" (UID: "a53b0913-db93-4105-ba34-5cce98fc8cc5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:22:11.260000 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:11.259956 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a53b0913-db93-4105-ba34-5cce98fc8cc5-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:22:11.260000 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:11.259977 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a53b0913-db93-4105-ba34-5cce98fc8cc5-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:22:11.260000 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:11.259991 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a53b0913-db93-4105-ba34-5cce98fc8cc5-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:22:11.262062 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:11.262012 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a53b0913-db93-4105-ba34-5cce98fc8cc5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a53b0913-db93-4105-ba34-5cce98fc8cc5" (UID: "a53b0913-db93-4105-ba34-5cce98fc8cc5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 19:22:11.262191 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:11.262107 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53b0913-db93-4105-ba34-5cce98fc8cc5-dshm" (OuterVolumeSpecName: "dshm") pod "a53b0913-db93-4105-ba34-5cce98fc8cc5" (UID: "a53b0913-db93-4105-ba34-5cce98fc8cc5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:22:11.262290 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:11.262169 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53b0913-db93-4105-ba34-5cce98fc8cc5-kube-api-access-drv76" (OuterVolumeSpecName: "kube-api-access-drv76") pod "a53b0913-db93-4105-ba34-5cce98fc8cc5" (UID: "a53b0913-db93-4105-ba34-5cce98fc8cc5"). InnerVolumeSpecName "kube-api-access-drv76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:22:11.304842 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:11.304808 2565 generic.go:356] "Generic (PLEG): container finished" podID="a53b0913-db93-4105-ba34-5cce98fc8cc5" containerID="1e54df18eb0ae1df4381c3cf28a236360660ae199849c081867fa97344dd517f" exitCode=0 Jul 31 19:22:11.304994 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:11.304853 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh" Jul 31 19:22:11.304994 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:11.304892 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh" event={"ID":"a53b0913-db93-4105-ba34-5cce98fc8cc5","Type":"ContainerDied","Data":"1e54df18eb0ae1df4381c3cf28a236360660ae199849c081867fa97344dd517f"} Jul 31 19:22:11.304994 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:11.304938 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh" event={"ID":"a53b0913-db93-4105-ba34-5cce98fc8cc5","Type":"ContainerDied","Data":"549ed42fc22d06aba68aeb51291938266baab29178fbe279354edc3afd4cb2bc"} Jul 31 19:22:11.304994 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:11.304961 2565 scope.go:117] "RemoveContainer" containerID="1e54df18eb0ae1df4381c3cf28a236360660ae199849c081867fa97344dd517f" Jul 31 19:22:11.314287 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:11.314269 2565 scope.go:117] "RemoveContainer" containerID="1e54df18eb0ae1df4381c3cf28a236360660ae199849c081867fa97344dd517f" Jul 31 19:22:11.314538 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:22:11.314518 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e54df18eb0ae1df4381c3cf28a236360660ae199849c081867fa97344dd517f\": container with ID starting with 1e54df18eb0ae1df4381c3cf28a236360660ae199849c081867fa97344dd517f not found: ID does not exist" containerID="1e54df18eb0ae1df4381c3cf28a236360660ae199849c081867fa97344dd517f" Jul 31 19:22:11.314598 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:11.314545 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e54df18eb0ae1df4381c3cf28a236360660ae199849c081867fa97344dd517f"} err="failed to get container status \"1e54df18eb0ae1df4381c3cf28a236360660ae199849c081867fa97344dd517f\": rpc error: code = NotFound desc = could not find container \"1e54df18eb0ae1df4381c3cf28a236360660ae199849c081867fa97344dd517f\": container with ID starting with 1e54df18eb0ae1df4381c3cf28a236360660ae199849c081867fa97344dd517f not found: ID does not exist" Jul 31 19:22:11.327472 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:11.327445 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh"] Jul 31 19:22:11.333796 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:11.333777 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-aaeacbf1/e2e-lifecycle-model-aaeacbf1-kserve-668ff89995-kxfrh"] Jul 31 19:22:11.360812 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:11.360790 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drv76\" (UniqueName: \"kubernetes.io/projected/a53b0913-db93-4105-ba34-5cce98fc8cc5-kube-api-access-drv76\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:22:11.360959 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:11.360814 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53b0913-db93-4105-ba34-5cce98fc8cc5-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:22:11.360959 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:11.360824 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a53b0913-db93-4105-ba34-5cce98fc8cc5-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:22:11.886645 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:11.886558 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a53b0913-db93-4105-ba34-5cce98fc8cc5" path="/var/lib/kubelet/pods/a53b0913-db93-4105-ba34-5cce98fc8cc5/volumes" Jul 31 19:22:21.924825 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:21.924789 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:22:33.464180 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:33.464145 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:22:33.929775 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:33.929742 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:22:39.910452 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:39.910376 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:22:39.974151 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:39.974112 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:22:40.179766 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:40.179679 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv"] Jul 31 19:22:40.179964 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:40.179938 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" podUID="da2bf329-b9d0-4c78-8e58-afa8bdef24bb" containerName="istio-proxy" containerID="cri-o://6d46d9141aad9c1dc11b4ad04b272e645f1dfef342d63c8fc1252011b9972ac7" gracePeriod=30 Jul 31 19:22:40.199933 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:40.199900 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:22:40.424820 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:40.424783 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:22:42.064163 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:42.064129 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:22:45.436050 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.436026 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:22:45.476195 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.476115 2565 generic.go:356] "Generic (PLEG): container finished" podID="da2bf329-b9d0-4c78-8e58-afa8bdef24bb" containerID="6d46d9141aad9c1dc11b4ad04b272e645f1dfef342d63c8fc1252011b9972ac7" exitCode=0 Jul 31 19:22:45.476195 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.476160 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" Jul 31 19:22:45.476195 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.476184 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" event={"ID":"da2bf329-b9d0-4c78-8e58-afa8bdef24bb","Type":"ContainerDied","Data":"6d46d9141aad9c1dc11b4ad04b272e645f1dfef342d63c8fc1252011b9972ac7"} Jul 31 19:22:45.476428 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.476262 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv" event={"ID":"da2bf329-b9d0-4c78-8e58-afa8bdef24bb","Type":"ContainerDied","Data":"9059a98f4b528374c9340ff1ab2de5a0f9d497bba0b44557e2084075cfa4de76"} Jul 31 19:22:45.476428 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.476281 2565 scope.go:117] "RemoveContainer" containerID="6d46d9141aad9c1dc11b4ad04b272e645f1dfef342d63c8fc1252011b9972ac7" Jul 31 19:22:45.483635 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.483613 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsp78\" (UniqueName: \"kubernetes.io/projected/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-kube-api-access-gsp78\") pod \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " Jul 31 19:22:45.483776 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.483650 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-istio-podinfo\") pod \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " Jul 31 19:22:45.483776 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.483683 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-credential-socket\") pod \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " Jul 31 19:22:45.483776 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.483708 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-istio-data\") pod \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " Jul 31 19:22:45.483776 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.483769 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-istio-envoy\") pod \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " Jul 31 19:22:45.483988 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.483795 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-workload-socket\") pod \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " Jul 31 19:22:45.484048 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.484013 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-istio-data" (OuterVolumeSpecName: "istio-data") pod "da2bf329-b9d0-4c78-8e58-afa8bdef24bb" (UID: "da2bf329-b9d0-4c78-8e58-afa8bdef24bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:22:45.484101 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.484080 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "da2bf329-b9d0-4c78-8e58-afa8bdef24bb" (UID: "da2bf329-b9d0-4c78-8e58-afa8bdef24bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:22:45.484298 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.484272 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "da2bf329-b9d0-4c78-8e58-afa8bdef24bb" (UID: "da2bf329-b9d0-4c78-8e58-afa8bdef24bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:22:45.484422 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.484301 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-workload-certs\") pod \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " Jul 31 19:22:45.484422 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.484350 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-istiod-ca-cert\") pod \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " Jul 31 19:22:45.484422 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.484386 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-istio-token\") pod \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\" (UID: \"da2bf329-b9d0-4c78-8e58-afa8bdef24bb\") " Jul 31 19:22:45.484600 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.484520 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "da2bf329-b9d0-4c78-8e58-afa8bdef24bb" (UID: "da2bf329-b9d0-4c78-8e58-afa8bdef24bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:22:45.484701 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.484680 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "da2bf329-b9d0-4c78-8e58-afa8bdef24bb" (UID: "da2bf329-b9d0-4c78-8e58-afa8bdef24bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 19:22:45.485062 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.485041 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-credential-socket\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:22:45.485169 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.485068 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-istio-data\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:22:45.485169 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.485084 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-workload-socket\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:22:45.485169 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.485097 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-workload-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:22:45.485169 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.485110 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-istiod-ca-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:22:45.485606 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.485583 2565 scope.go:117] "RemoveContainer" containerID="18f7330001e02f2918193f71fd78803e17cc6de7e63bc2dc280208d84e648514" Jul 31 19:22:45.486266 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.486188 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "da2bf329-b9d0-4c78-8e58-afa8bdef24bb" (UID: "da2bf329-b9d0-4c78-8e58-afa8bdef24bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 19:22:45.486266 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.486214 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-kube-api-access-gsp78" (OuterVolumeSpecName: "kube-api-access-gsp78") pod "da2bf329-b9d0-4c78-8e58-afa8bdef24bb" (UID: "da2bf329-b9d0-4c78-8e58-afa8bdef24bb"). InnerVolumeSpecName "kube-api-access-gsp78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:22:45.486836 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.486808 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "da2bf329-b9d0-4c78-8e58-afa8bdef24bb" (UID: "da2bf329-b9d0-4c78-8e58-afa8bdef24bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:22:45.486972 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.486948 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-istio-token" (OuterVolumeSpecName: "istio-token") pod "da2bf329-b9d0-4c78-8e58-afa8bdef24bb" (UID: "da2bf329-b9d0-4c78-8e58-afa8bdef24bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:22:45.504873 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.504855 2565 scope.go:117] "RemoveContainer" containerID="6d46d9141aad9c1dc11b4ad04b272e645f1dfef342d63c8fc1252011b9972ac7" Jul 31 19:22:45.505140 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:22:45.505122 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d46d9141aad9c1dc11b4ad04b272e645f1dfef342d63c8fc1252011b9972ac7\": container with ID starting with 6d46d9141aad9c1dc11b4ad04b272e645f1dfef342d63c8fc1252011b9972ac7 not found: ID does not exist" containerID="6d46d9141aad9c1dc11b4ad04b272e645f1dfef342d63c8fc1252011b9972ac7" Jul 31 19:22:45.505194 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.505149 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d46d9141aad9c1dc11b4ad04b272e645f1dfef342d63c8fc1252011b9972ac7"} err="failed to get container status \"6d46d9141aad9c1dc11b4ad04b272e645f1dfef342d63c8fc1252011b9972ac7\": rpc error: code = NotFound desc = could not find container \"6d46d9141aad9c1dc11b4ad04b272e645f1dfef342d63c8fc1252011b9972ac7\": container with ID starting with 6d46d9141aad9c1dc11b4ad04b272e645f1dfef342d63c8fc1252011b9972ac7 not found: ID does not exist" Jul 31 19:22:45.505194 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.505167 2565 scope.go:117] "RemoveContainer" containerID="18f7330001e02f2918193f71fd78803e17cc6de7e63bc2dc280208d84e648514" Jul 31 19:22:45.505426 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:22:45.505407 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18f7330001e02f2918193f71fd78803e17cc6de7e63bc2dc280208d84e648514\": container with ID starting with 18f7330001e02f2918193f71fd78803e17cc6de7e63bc2dc280208d84e648514 not found: ID does not exist" containerID="18f7330001e02f2918193f71fd78803e17cc6de7e63bc2dc280208d84e648514" Jul 31 19:22:45.505469 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.505433 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f7330001e02f2918193f71fd78803e17cc6de7e63bc2dc280208d84e648514"} err="failed to get container status \"18f7330001e02f2918193f71fd78803e17cc6de7e63bc2dc280208d84e648514\": rpc error: code = NotFound desc = could not find container \"18f7330001e02f2918193f71fd78803e17cc6de7e63bc2dc280208d84e648514\": container with ID starting with 18f7330001e02f2918193f71fd78803e17cc6de7e63bc2dc280208d84e648514 not found: ID does not exist" Jul 31 19:22:45.585630 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.585601 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-istio-podinfo\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:22:45.585630 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.585625 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-istio-envoy\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:22:45.585630 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.585635 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-istio-token\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:22:45.585858 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.585644 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsp78\" (UniqueName: \"kubernetes.io/projected/da2bf329-b9d0-4c78-8e58-afa8bdef24bb-kube-api-access-gsp78\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:22:45.803199 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.803170 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv"] Jul 31 19:22:45.820177 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.820147 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-aaeacbf1-openshift-default-bcbfdbdd8-zrtkv"] Jul 31 19:22:45.887208 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:45.887176 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da2bf329-b9d0-4c78-8e58-afa8bdef24bb" path="/var/lib/kubelet/pods/da2bf329-b9d0-4c78-8e58-afa8bdef24bb/volumes" Jul 31 19:22:51.749385 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:51.749349 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:22:51.831791 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:51.831746 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:22:52.522167 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:52.522132 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:22:52.610757 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:52.610721 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:22:52.743134 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:52.743101 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:22:53.948206 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:53.948172 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:22:54.380680 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:22:54.380644 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:23:00.929898 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:00.929864 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:23:01.028516 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:01.028481 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:23:01.618805 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:01.618756 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:23:06.956682 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:06.956646 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:23:07.836542 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:07.836507 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:23:07.942648 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:07.942612 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:23:08.037387 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:08.037352 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:23:18.506394 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:18.506356 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:23:18.714349 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:18.714314 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:23:35.182565 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:35.182509 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:23:36.496885 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:36.496849 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:23:39.617594 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:39.617557 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:23:46.289087 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:46.289053 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:23:46.362493 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:46.362459 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:23:46.621785 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:46.621699 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:23:46.905555 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:46.905480 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l"] Jul 31 19:23:46.905878 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:46.905865 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da2bf329-b9d0-4c78-8e58-afa8bdef24bb" containerName="istio-proxy" Jul 31 19:23:46.905936 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:46.905879 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="da2bf329-b9d0-4c78-8e58-afa8bdef24bb" containerName="istio-proxy" Jul 31 19:23:46.905936 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:46.905890 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a53b0913-db93-4105-ba34-5cce98fc8cc5" containerName="main" Jul 31 19:23:46.905936 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:46.905897 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a53b0913-db93-4105-ba34-5cce98fc8cc5" containerName="main" Jul 31 19:23:46.905936 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:46.905916 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da2bf329-b9d0-4c78-8e58-afa8bdef24bb" containerName="istio-proxy" Jul 31 19:23:46.905936 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:46.905922 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="da2bf329-b9d0-4c78-8e58-afa8bdef24bb" containerName="istio-proxy" Jul 31 19:23:46.906095 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:46.905984 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="da2bf329-b9d0-4c78-8e58-afa8bdef24bb" containerName="istio-proxy" Jul 31 19:23:46.906095 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:46.905994 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="da2bf329-b9d0-4c78-8e58-afa8bdef24bb" containerName="istio-proxy" Jul 31 19:23:46.906095 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:46.906000 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a53b0913-db93-4105-ba34-5cce98fc8cc5" containerName="main" Jul 31 19:23:46.910080 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:46.910062 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:46.912145 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:46.912125 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-66ce83-openshift-default-dockercfg-rzj8x\"" Jul 31 19:23:46.921910 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:46.921889 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l"] Jul 31 19:23:46.925702 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:46.925679 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:23:46.968912 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:46.968885 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/050f3f0b-8496-4f94-a881-19d1598ab6cc-istio-token\") pod \"e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:46.969019 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:46.968919 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/050f3f0b-8496-4f94-a881-19d1598ab6cc-istio-podinfo\") pod \"e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:46.969019 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:46.968940 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/050f3f0b-8496-4f94-a881-19d1598ab6cc-credential-socket\") pod \"e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:46.969019 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:46.969007 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/050f3f0b-8496-4f94-a881-19d1598ab6cc-workload-certs\") pod \"e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:46.969133 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:46.969039 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/050f3f0b-8496-4f94-a881-19d1598ab6cc-istio-envoy\") pod \"e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:46.969133 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:46.969070 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/050f3f0b-8496-4f94-a881-19d1598ab6cc-istio-data\") pod \"e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:46.969133 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:46.969088 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/050f3f0b-8496-4f94-a881-19d1598ab6cc-istiod-ca-cert\") pod \"e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:46.969326 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:46.969306 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/050f3f0b-8496-4f94-a881-19d1598ab6cc-workload-socket\") pod \"e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:46.969745 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:46.969568 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q69gl\" (UniqueName: \"kubernetes.io/projected/050f3f0b-8496-4f94-a881-19d1598ab6cc-kube-api-access-q69gl\") pod \"e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:47.070575 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:47.070543 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/050f3f0b-8496-4f94-a881-19d1598ab6cc-workload-socket\") pod \"e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:47.070750 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:47.070601 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q69gl\" (UniqueName: \"kubernetes.io/projected/050f3f0b-8496-4f94-a881-19d1598ab6cc-kube-api-access-q69gl\") pod \"e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:47.070750 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:47.070624 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/050f3f0b-8496-4f94-a881-19d1598ab6cc-istio-token\") pod \"e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:47.070750 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:47.070645 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/050f3f0b-8496-4f94-a881-19d1598ab6cc-istio-podinfo\") pod \"e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:47.070750 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:47.070668 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/050f3f0b-8496-4f94-a881-19d1598ab6cc-credential-socket\") pod \"e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:47.070750 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:47.070689 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/050f3f0b-8496-4f94-a881-19d1598ab6cc-workload-certs\") pod \"e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:47.070750 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:47.070708 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/050f3f0b-8496-4f94-a881-19d1598ab6cc-istio-envoy\") pod \"e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:47.070750 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:47.070726 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/050f3f0b-8496-4f94-a881-19d1598ab6cc-istio-data\") pod \"e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:47.071140 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:47.070999 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/050f3f0b-8496-4f94-a881-19d1598ab6cc-workload-socket\") pod \"e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:47.071140 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:47.071096 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/050f3f0b-8496-4f94-a881-19d1598ab6cc-credential-socket\") pod \"e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:47.071140 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:47.071117 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/050f3f0b-8496-4f94-a881-19d1598ab6cc-istio-data\") pod \"e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:47.071316 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:47.071002 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/050f3f0b-8496-4f94-a881-19d1598ab6cc-istiod-ca-cert\") pod \"e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:47.071316 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:47.071255 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/050f3f0b-8496-4f94-a881-19d1598ab6cc-workload-certs\") pod \"e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:47.071620 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:47.071602 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/050f3f0b-8496-4f94-a881-19d1598ab6cc-istiod-ca-cert\") pod \"e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:47.073101 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:47.073067 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/050f3f0b-8496-4f94-a881-19d1598ab6cc-istio-envoy\") pod \"e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:47.073264 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:47.073227 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/050f3f0b-8496-4f94-a881-19d1598ab6cc-istio-podinfo\") pod \"e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:47.084285 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:47.084259 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/050f3f0b-8496-4f94-a881-19d1598ab6cc-istio-token\") pod \"e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:47.084627 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:47.084610 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q69gl\" (UniqueName: \"kubernetes.io/projected/050f3f0b-8496-4f94-a881-19d1598ab6cc-kube-api-access-q69gl\") pod \"e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:47.220267 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:47.220176 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:47.249082 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:47.249052 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:23:47.362424 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:47.362319 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l"] Jul 31 19:23:47.365455 ip-10-0-139-39 kubenswrapper[2565]: W0731 19:23:47.365419 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod050f3f0b_8496_4f94_a881_19d1598ab6cc.slice/crio-e350afa354ebb1029621e15b96af5510a485db9bfdec2d093c3de13442e5c2fb WatchSource:0}: Error finding container e350afa354ebb1029621e15b96af5510a485db9bfdec2d093c3de13442e5c2fb: Status 404 returned error can't find the container with id e350afa354ebb1029621e15b96af5510a485db9bfdec2d093c3de13442e5c2fb Jul 31 19:23:47.367718 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:47.367683 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:23:47.367841 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:47.367747 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:23:47.367841 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:47.367782 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:23:47.792722 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:47.792680 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" event={"ID":"050f3f0b-8496-4f94-a881-19d1598ab6cc","Type":"ContainerStarted","Data":"850f417d2208de9116c14ca4a5927d5a424607848294199d1f7ee43c1bdd4d62"} Jul 31 19:23:47.792905 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:47.792765 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" event={"ID":"050f3f0b-8496-4f94-a881-19d1598ab6cc","Type":"ContainerStarted","Data":"e350afa354ebb1029621e15b96af5510a485db9bfdec2d093c3de13442e5c2fb"} Jul 31 19:23:47.818422 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:47.818373 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" podStartSLOduration=1.8183595289999999 podStartE2EDuration="1.818359529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:23:47.816856932 +0000 UTC m=+2404.454846733" watchObservedRunningTime="2026-07-31 19:23:47.818359529 +0000 UTC m=+2404.456349329" Jul 31 19:23:48.220628 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:48.220521 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:48.225650 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:48.225626 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:48.797628 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:48.797589 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:48.798526 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:48.798509 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:23:49.757066 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:49.757030 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:23:49.855760 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:49.855725 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:23:50.224961 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:50.224868 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:23:50.336231 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:50.336196 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:23:51.028650 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:51.028614 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:23:51.381765 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:51.381686 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:23:51.722273 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:51.722161 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:23:51.820118 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:51.820085 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:23:53.665636 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:23:53.665600 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:24:03.799630 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:03.799592 2565 patch_prober.go:28] interesting pod/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 31 19:24:03.800062 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:03.799650 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" podUID="050f3f0b-8496-4f94-a881-19d1598ab6cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 31 19:24:03.880394 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:03.880355 2565 generic.go:356] "Generic (PLEG): container finished" podID="050f3f0b-8496-4f94-a881-19d1598ab6cc" containerID="850f417d2208de9116c14ca4a5927d5a424607848294199d1f7ee43c1bdd4d62" exitCode=0 Jul 31 19:24:03.886445 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:03.886415 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" event={"ID":"050f3f0b-8496-4f94-a881-19d1598ab6cc","Type":"ContainerDied","Data":"850f417d2208de9116c14ca4a5927d5a424607848294199d1f7ee43c1bdd4d62"} Jul 31 19:24:03.886791 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:03.886774 2565 scope.go:117] "RemoveContainer" containerID="850f417d2208de9116c14ca4a5927d5a424607848294199d1f7ee43c1bdd4d62" Jul 31 19:24:03.887825 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:03.887797 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:24:03.887877 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:03.887869 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:24:03.887913 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:03.887901 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:24:04.886820 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:04.886786 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" event={"ID":"050f3f0b-8496-4f94-a881-19d1598ab6cc","Type":"ContainerStarted","Data":"21e45f105d1b40e4ce76cfcaa97d7fd58581bc4026c95c5c79bd895172d93250"} Jul 31 19:24:05.220907 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:05.220812 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:24:05.221165 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:05.221146 2565 patch_prober.go:28] interesting pod/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 31 19:24:05.221217 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:05.221202 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" podUID="050f3f0b-8496-4f94-a881-19d1598ab6cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 31 19:24:06.220689 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:06.220647 2565 patch_prober.go:28] interesting pod/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 31 19:24:06.221089 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:06.220705 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" podUID="050f3f0b-8496-4f94-a881-19d1598ab6cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 31 19:24:07.221121 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:07.221084 2565 patch_prober.go:28] interesting pod/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 31 19:24:07.221526 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:07.221139 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" podUID="050f3f0b-8496-4f94-a881-19d1598ab6cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 31 19:24:08.220870 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:08.220836 2565 patch_prober.go:28] interesting pod/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 31 19:24:08.221049 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:08.220897 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" podUID="050f3f0b-8496-4f94-a881-19d1598ab6cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 31 19:24:09.221407 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:09.221366 2565 patch_prober.go:28] interesting pod/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 31 19:24:09.221882 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:09.221433 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" podUID="050f3f0b-8496-4f94-a881-19d1598ab6cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 31 19:24:10.224574 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:10.224541 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:24:10.225023 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:10.224812 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:24:10.225622 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:10.225605 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:24:14.715937 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:14.715902 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:24:15.047285 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.047186 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:24:15.311258 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.311168 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7"] Jul 31 19:24:15.314282 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.314230 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:15.317604 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.317582 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8aa450-openshift-default-dockercfg-kv9tz\"" Jul 31 19:24:15.325406 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.325384 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7"] Jul 31 19:24:15.347821 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.347795 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/112c8c10-517d-4069-a825-8783c6d683a7-istiod-ca-cert\") pod \"e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:15.347936 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.347830 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/112c8c10-517d-4069-a825-8783c6d683a7-istio-envoy\") pod \"e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:15.347936 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.347874 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/112c8c10-517d-4069-a825-8783c6d683a7-istio-data\") pod \"e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:15.348028 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.347953 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/112c8c10-517d-4069-a825-8783c6d683a7-credential-socket\") pod \"e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:15.348028 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.347983 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x89js\" (UniqueName: \"kubernetes.io/projected/112c8c10-517d-4069-a825-8783c6d683a7-kube-api-access-x89js\") pod \"e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:15.348028 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.348006 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/112c8c10-517d-4069-a825-8783c6d683a7-workload-socket\") pod \"e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:15.348155 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.348071 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/112c8c10-517d-4069-a825-8783c6d683a7-workload-certs\") pod \"e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:15.348155 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.348118 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/112c8c10-517d-4069-a825-8783c6d683a7-istio-token\") pod \"e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:15.348155 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.348141 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/112c8c10-517d-4069-a825-8783c6d683a7-istio-podinfo\") pod \"e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:15.349827 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.349800 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:24:15.449064 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.449030 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/112c8c10-517d-4069-a825-8783c6d683a7-istio-token\") pod \"e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:15.449259 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.449080 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/112c8c10-517d-4069-a825-8783c6d683a7-istio-podinfo\") pod \"e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:15.449259 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.449124 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/112c8c10-517d-4069-a825-8783c6d683a7-istiod-ca-cert\") pod \"e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:15.449259 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.449156 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/112c8c10-517d-4069-a825-8783c6d683a7-istio-envoy\") pod \"e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:15.449461 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.449409 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/112c8c10-517d-4069-a825-8783c6d683a7-istio-data\") pod \"e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:15.449542 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.449526 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/112c8c10-517d-4069-a825-8783c6d683a7-credential-socket\") pod \"e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:15.449605 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.449562 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x89js\" (UniqueName: \"kubernetes.io/projected/112c8c10-517d-4069-a825-8783c6d683a7-kube-api-access-x89js\") pod \"e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:15.449605 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.449596 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/112c8c10-517d-4069-a825-8783c6d683a7-workload-socket\") pod \"e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:15.449705 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.449635 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/112c8c10-517d-4069-a825-8783c6d683a7-workload-certs\") pod \"e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:15.449793 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.449767 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/112c8c10-517d-4069-a825-8783c6d683a7-istio-data\") pod \"e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:15.449968 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.449943 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/112c8c10-517d-4069-a825-8783c6d683a7-workload-socket\") pod \"e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:15.450045 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.449865 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/112c8c10-517d-4069-a825-8783c6d683a7-credential-socket\") pod \"e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:15.450045 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.449997 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/112c8c10-517d-4069-a825-8783c6d683a7-istiod-ca-cert\") pod \"e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:15.450045 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.450012 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/112c8c10-517d-4069-a825-8783c6d683a7-workload-certs\") pod \"e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:15.451679 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.451657 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/112c8c10-517d-4069-a825-8783c6d683a7-istio-envoy\") pod \"e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:15.451920 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.451899 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/112c8c10-517d-4069-a825-8783c6d683a7-istio-podinfo\") pod \"e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:15.457483 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.457459 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/112c8c10-517d-4069-a825-8783c6d683a7-istio-token\") pod \"e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:15.457753 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.457736 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x89js\" (UniqueName: \"kubernetes.io/projected/112c8c10-517d-4069-a825-8783c6d683a7-kube-api-access-x89js\") pod \"e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:15.624859 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.624780 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:15.768503 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.768477 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7"] Jul 31 19:24:15.770576 ip-10-0-139-39 kubenswrapper[2565]: W0731 19:24:15.770548 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod112c8c10_517d_4069_a825_8783c6d683a7.slice/crio-8da1fe46bfcb978b86dcec2ebccaadeff223ddf5f2516647c76f55a3bddd8b6a WatchSource:0}: Error finding container 8da1fe46bfcb978b86dcec2ebccaadeff223ddf5f2516647c76f55a3bddd8b6a: Status 404 returned error can't find the container with id 8da1fe46bfcb978b86dcec2ebccaadeff223ddf5f2516647c76f55a3bddd8b6a Jul 31 19:24:15.772743 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.772708 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:24:15.772864 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.772775 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:24:15.772864 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.772804 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:24:15.899498 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.899416 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:24:15.943138 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.943099 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" event={"ID":"112c8c10-517d-4069-a825-8783c6d683a7","Type":"ContainerStarted","Data":"8ea65fc3fd37e1ee82120d832b4055f216d8510e03cec387726d48ece94e0bab"} Jul 31 19:24:15.943348 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.943164 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" event={"ID":"112c8c10-517d-4069-a825-8783c6d683a7","Type":"ContainerStarted","Data":"8da1fe46bfcb978b86dcec2ebccaadeff223ddf5f2516647c76f55a3bddd8b6a"} Jul 31 19:24:15.965113 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:15.965057 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" podStartSLOduration=0.965036429 podStartE2EDuration="965.036429ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:24:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:24:15.962811541 +0000 UTC m=+2432.600801341" watchObservedRunningTime="2026-07-31 19:24:15.965036429 +0000 UTC m=+2432.603026229" Jul 31 19:24:16.625736 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:16.625692 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:16.630713 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:16.630687 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:16.948229 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:16.948132 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:16.949203 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:16.949179 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:17.957779 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:17.957740 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:24:18.050083 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:18.050051 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:24:19.098089 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:19.098058 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:24:19.468635 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:19.468542 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:24:19.645877 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:19.645846 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:24:19.743977 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:19.743896 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:24:22.158890 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:22.158846 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:24:31.949857 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:31.949756 2565 patch_prober.go:28] interesting pod/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 31 19:24:31.949857 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:31.949811 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" podUID="112c8c10-517d-4069-a825-8783c6d683a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 31 19:24:32.027489 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:32.027440 2565 generic.go:356] "Generic (PLEG): container finished" podID="112c8c10-517d-4069-a825-8783c6d683a7" containerID="8ea65fc3fd37e1ee82120d832b4055f216d8510e03cec387726d48ece94e0bab" exitCode=0 Jul 31 19:24:32.027677 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:32.027549 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" event={"ID":"112c8c10-517d-4069-a825-8783c6d683a7","Type":"ContainerDied","Data":"8ea65fc3fd37e1ee82120d832b4055f216d8510e03cec387726d48ece94e0bab"} Jul 31 19:24:32.028230 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:32.028198 2565 scope.go:117] "RemoveContainer" containerID="8ea65fc3fd37e1ee82120d832b4055f216d8510e03cec387726d48ece94e0bab" Jul 31 19:24:32.029841 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:32.029806 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:24:32.029908 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:32.029893 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:24:32.029969 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:32.029955 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:24:33.037866 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:33.037766 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" event={"ID":"112c8c10-517d-4069-a825-8783c6d683a7","Type":"ContainerStarted","Data":"0a73315d8fe3a176f7bc84e0997c40ca68900b2b618565ea97f8adc73fed7f78"} Jul 31 19:24:33.625159 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:33.625122 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:33.625435 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:33.625413 2565 patch_prober.go:28] interesting pod/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 31 19:24:33.625516 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:33.625471 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" podUID="112c8c10-517d-4069-a825-8783c6d683a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 31 19:24:34.625735 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:34.625695 2565 patch_prober.go:28] interesting pod/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 31 19:24:34.626142 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:34.625752 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" podUID="112c8c10-517d-4069-a825-8783c6d683a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 31 19:24:35.625534 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:35.625500 2565 patch_prober.go:28] interesting pod/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 31 19:24:35.625704 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:35.625554 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" podUID="112c8c10-517d-4069-a825-8783c6d683a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 31 19:24:36.625411 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:36.625374 2565 patch_prober.go:28] interesting pod/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 31 19:24:36.625805 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:36.625437 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" podUID="112c8c10-517d-4069-a825-8783c6d683a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 31 19:24:37.625724 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:37.625687 2565 patch_prober.go:28] interesting pod/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 31 19:24:37.626142 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:37.625772 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" podUID="112c8c10-517d-4069-a825-8783c6d683a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 31 19:24:38.629460 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:38.629431 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:38.629852 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:38.629779 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:38.630342 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:38.630326 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:24:42.809824 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:42.809792 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:24:47.573053 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:47.573009 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn"] Jul 31 19:24:47.576282 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:47.576232 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn" Jul 31 19:24:47.579138 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:47.579112 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-66ce83\"/\"default-dockercfg-js6z6\"" Jul 31 19:24:47.579354 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:47.579327 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-66ce83\"/\"openshift-service-ca.crt\"" Jul 31 19:24:47.579510 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:47.579200 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-66ce83\"/\"test-model-66ce83-kserve-self-signed-certs\"" Jul 31 19:24:47.579632 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:47.579391 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-66ce83\"/\"kube-root-ca.crt\"" Jul 31 19:24:47.586273 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:47.586231 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn"] Jul 31 19:24:47.599985 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:47.599947 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:24:47.680967 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:47.680932 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a774d95-94f5-4ff3-ab69-0f32b7aba847-tls-certs\") pod \"test-model-66ce83-kserve-844478dcff-s9tcn\" (UID: \"9a774d95-94f5-4ff3-ab69-0f32b7aba847\") " pod="ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn" Jul 31 19:24:47.681153 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:47.680999 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a774d95-94f5-4ff3-ab69-0f32b7aba847-tmp-dir\") pod \"test-model-66ce83-kserve-844478dcff-s9tcn\" (UID: \"9a774d95-94f5-4ff3-ab69-0f32b7aba847\") " pod="ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn" Jul 31 19:24:47.681153 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:47.681030 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a774d95-94f5-4ff3-ab69-0f32b7aba847-model-cache\") pod \"test-model-66ce83-kserve-844478dcff-s9tcn\" (UID: \"9a774d95-94f5-4ff3-ab69-0f32b7aba847\") " pod="ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn" Jul 31 19:24:47.681153 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:47.681083 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a774d95-94f5-4ff3-ab69-0f32b7aba847-dshm\") pod \"test-model-66ce83-kserve-844478dcff-s9tcn\" (UID: \"9a774d95-94f5-4ff3-ab69-0f32b7aba847\") " pod="ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn" Jul 31 19:24:47.681153 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:47.681139 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a774d95-94f5-4ff3-ab69-0f32b7aba847-home\") pod \"test-model-66ce83-kserve-844478dcff-s9tcn\" (UID: \"9a774d95-94f5-4ff3-ab69-0f32b7aba847\") " pod="ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn" Jul 31 19:24:47.681382 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:47.681211 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n6x6\" (UniqueName: \"kubernetes.io/projected/9a774d95-94f5-4ff3-ab69-0f32b7aba847-kube-api-access-2n6x6\") pod \"test-model-66ce83-kserve-844478dcff-s9tcn\" (UID: \"9a774d95-94f5-4ff3-ab69-0f32b7aba847\") " pod="ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn" Jul 31 19:24:47.782282 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:47.782233 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n6x6\" (UniqueName: \"kubernetes.io/projected/9a774d95-94f5-4ff3-ab69-0f32b7aba847-kube-api-access-2n6x6\") pod \"test-model-66ce83-kserve-844478dcff-s9tcn\" (UID: \"9a774d95-94f5-4ff3-ab69-0f32b7aba847\") " pod="ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn" Jul 31 19:24:47.782474 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:47.782312 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a774d95-94f5-4ff3-ab69-0f32b7aba847-tls-certs\") pod \"test-model-66ce83-kserve-844478dcff-s9tcn\" (UID: \"9a774d95-94f5-4ff3-ab69-0f32b7aba847\") " pod="ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn" Jul 31 19:24:47.782474 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:47.782348 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a774d95-94f5-4ff3-ab69-0f32b7aba847-tmp-dir\") pod \"test-model-66ce83-kserve-844478dcff-s9tcn\" (UID: \"9a774d95-94f5-4ff3-ab69-0f32b7aba847\") " pod="ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn" Jul 31 19:24:47.782474 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:47.782367 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a774d95-94f5-4ff3-ab69-0f32b7aba847-model-cache\") pod \"test-model-66ce83-kserve-844478dcff-s9tcn\" (UID: \"9a774d95-94f5-4ff3-ab69-0f32b7aba847\") " pod="ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn" Jul 31 19:24:47.782474 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:47.782413 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a774d95-94f5-4ff3-ab69-0f32b7aba847-dshm\") pod \"test-model-66ce83-kserve-844478dcff-s9tcn\" (UID: \"9a774d95-94f5-4ff3-ab69-0f32b7aba847\") " pod="ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn" Jul 31 19:24:47.782474 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:47.782447 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a774d95-94f5-4ff3-ab69-0f32b7aba847-home\") pod \"test-model-66ce83-kserve-844478dcff-s9tcn\" (UID: \"9a774d95-94f5-4ff3-ab69-0f32b7aba847\") " pod="ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn" Jul 31 19:24:47.782856 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:47.782831 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a774d95-94f5-4ff3-ab69-0f32b7aba847-model-cache\") pod \"test-model-66ce83-kserve-844478dcff-s9tcn\" (UID: \"9a774d95-94f5-4ff3-ab69-0f32b7aba847\") " pod="ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn" Jul 31 19:24:47.782919 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:47.782856 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a774d95-94f5-4ff3-ab69-0f32b7aba847-tmp-dir\") pod \"test-model-66ce83-kserve-844478dcff-s9tcn\" (UID: \"9a774d95-94f5-4ff3-ab69-0f32b7aba847\") " pod="ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn" Jul 31 19:24:47.782955 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:47.782922 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a774d95-94f5-4ff3-ab69-0f32b7aba847-home\") pod \"test-model-66ce83-kserve-844478dcff-s9tcn\" (UID: \"9a774d95-94f5-4ff3-ab69-0f32b7aba847\") " pod="ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn" Jul 31 19:24:47.784709 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:47.784684 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a774d95-94f5-4ff3-ab69-0f32b7aba847-dshm\") pod \"test-model-66ce83-kserve-844478dcff-s9tcn\" (UID: \"9a774d95-94f5-4ff3-ab69-0f32b7aba847\") " pod="ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn" Jul 31 19:24:47.784937 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:47.784917 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a774d95-94f5-4ff3-ab69-0f32b7aba847-tls-certs\") pod \"test-model-66ce83-kserve-844478dcff-s9tcn\" (UID: \"9a774d95-94f5-4ff3-ab69-0f32b7aba847\") " pod="ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn" Jul 31 19:24:47.791362 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:47.791342 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n6x6\" (UniqueName: \"kubernetes.io/projected/9a774d95-94f5-4ff3-ab69-0f32b7aba847-kube-api-access-2n6x6\") pod \"test-model-66ce83-kserve-844478dcff-s9tcn\" (UID: \"9a774d95-94f5-4ff3-ab69-0f32b7aba847\") " pod="ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn" Jul 31 19:24:47.889466 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:47.889385 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn" Jul 31 19:24:48.032446 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:48.032347 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn"] Jul 31 19:24:48.034618 ip-10-0-139-39 kubenswrapper[2565]: W0731 19:24:48.034586 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a774d95_94f5_4ff3_ab69_0f32b7aba847.slice/crio-345407ca38595ba33074a243ba75343078d503ba18891c666f987bf6743dbb9b WatchSource:0}: Error finding container 345407ca38595ba33074a243ba75343078d503ba18891c666f987bf6743dbb9b: Status 404 returned error can't find the container with id 345407ca38595ba33074a243ba75343078d503ba18891c666f987bf6743dbb9b Jul 31 19:24:48.116497 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:48.116465 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn" event={"ID":"9a774d95-94f5-4ff3-ab69-0f32b7aba847","Type":"ContainerStarted","Data":"08aefd85713a649d11294f8d46eb95a1fe631245e259e9af13c22e3e396a3709"} Jul 31 19:24:48.116693 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:48.116505 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn" event={"ID":"9a774d95-94f5-4ff3-ab69-0f32b7aba847","Type":"ContainerStarted","Data":"345407ca38595ba33074a243ba75343078d503ba18891c666f987bf6743dbb9b"} Jul 31 19:24:48.135563 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:48.135516 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn" podStartSLOduration=1.135500327 podStartE2EDuration="1.135500327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:24:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:24:48.133436347 +0000 UTC m=+2464.771426157" watchObservedRunningTime="2026-07-31 19:24:48.135500327 +0000 UTC m=+2464.773490126" Jul 31 19:24:51.066380 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:51.066337 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:24:57.889738 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:57.889703 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn" Jul 31 19:24:57.890196 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:57.889915 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn" Jul 31 19:24:57.902143 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:57.902117 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn" Jul 31 19:24:58.181481 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:24:58.181389 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn" Jul 31 19:25:13.818610 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:25:13.818574 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:25:17.661028 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:25:17.660992 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:25:57.172570 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:25:57.172494 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn"] Jul 31 19:25:57.195988 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:25:57.195944 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:25:58.105001 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:25:58.104967 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:25:58.379767 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:25:58.379670 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:26:04.557771 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:04.557725 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:26:08.822130 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:08.822097 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:26:08.895061 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:08.895020 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:26:08.964559 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:08.964527 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:26:09.025887 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:09.025847 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:26:09.388717 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:09.388670 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:26:10.868641 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:10.868601 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:26:11.373603 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:11.373561 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:26:12.150115 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:12.150079 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:26:12.191737 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:12.191685 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn" podUID="9a774d95-94f5-4ff3-ab69-0f32b7aba847" containerName="main" containerID="cri-o://08aefd85713a649d11294f8d46eb95a1fe631245e259e9af13c22e3e396a3709" gracePeriod=45 Jul 31 19:26:12.453534 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:12.453504 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn" Jul 31 19:26:12.543214 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:12.543182 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a774d95-94f5-4ff3-ab69-0f32b7aba847-dshm\") pod \"9a774d95-94f5-4ff3-ab69-0f32b7aba847\" (UID: \"9a774d95-94f5-4ff3-ab69-0f32b7aba847\") " Jul 31 19:26:12.543409 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:12.543227 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2n6x6\" (UniqueName: \"kubernetes.io/projected/9a774d95-94f5-4ff3-ab69-0f32b7aba847-kube-api-access-2n6x6\") pod \"9a774d95-94f5-4ff3-ab69-0f32b7aba847\" (UID: \"9a774d95-94f5-4ff3-ab69-0f32b7aba847\") " Jul 31 19:26:12.543409 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:12.543272 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a774d95-94f5-4ff3-ab69-0f32b7aba847-tmp-dir\") pod \"9a774d95-94f5-4ff3-ab69-0f32b7aba847\" (UID: \"9a774d95-94f5-4ff3-ab69-0f32b7aba847\") " Jul 31 19:26:12.543409 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:12.543294 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a774d95-94f5-4ff3-ab69-0f32b7aba847-tls-certs\") pod \"9a774d95-94f5-4ff3-ab69-0f32b7aba847\" (UID: \"9a774d95-94f5-4ff3-ab69-0f32b7aba847\") " Jul 31 19:26:12.543409 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:12.543310 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a774d95-94f5-4ff3-ab69-0f32b7aba847-model-cache\") pod \"9a774d95-94f5-4ff3-ab69-0f32b7aba847\" (UID: \"9a774d95-94f5-4ff3-ab69-0f32b7aba847\") " Jul 31 19:26:12.543409 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:12.543324 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a774d95-94f5-4ff3-ab69-0f32b7aba847-home\") pod \"9a774d95-94f5-4ff3-ab69-0f32b7aba847\" (UID: \"9a774d95-94f5-4ff3-ab69-0f32b7aba847\") " Jul 31 19:26:12.543683 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:12.543620 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a774d95-94f5-4ff3-ab69-0f32b7aba847-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9a774d95-94f5-4ff3-ab69-0f32b7aba847" (UID: "9a774d95-94f5-4ff3-ab69-0f32b7aba847"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:26:12.543683 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:12.543660 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a774d95-94f5-4ff3-ab69-0f32b7aba847-model-cache" (OuterVolumeSpecName: "model-cache") pod "9a774d95-94f5-4ff3-ab69-0f32b7aba847" (UID: "9a774d95-94f5-4ff3-ab69-0f32b7aba847"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:26:12.543797 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:12.543756 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a774d95-94f5-4ff3-ab69-0f32b7aba847-home" (OuterVolumeSpecName: "home") pod "9a774d95-94f5-4ff3-ab69-0f32b7aba847" (UID: "9a774d95-94f5-4ff3-ab69-0f32b7aba847"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:26:12.543853 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:12.543829 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a774d95-94f5-4ff3-ab69-0f32b7aba847-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:26:12.543902 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:12.543851 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a774d95-94f5-4ff3-ab69-0f32b7aba847-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:26:12.545567 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:12.545541 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a774d95-94f5-4ff3-ab69-0f32b7aba847-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9a774d95-94f5-4ff3-ab69-0f32b7aba847" (UID: "9a774d95-94f5-4ff3-ab69-0f32b7aba847"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 19:26:12.545802 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:12.545788 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a774d95-94f5-4ff3-ab69-0f32b7aba847-dshm" (OuterVolumeSpecName: "dshm") pod "9a774d95-94f5-4ff3-ab69-0f32b7aba847" (UID: "9a774d95-94f5-4ff3-ab69-0f32b7aba847"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:26:12.545862 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:12.545824 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a774d95-94f5-4ff3-ab69-0f32b7aba847-kube-api-access-2n6x6" (OuterVolumeSpecName: "kube-api-access-2n6x6") pod "9a774d95-94f5-4ff3-ab69-0f32b7aba847" (UID: "9a774d95-94f5-4ff3-ab69-0f32b7aba847"). InnerVolumeSpecName "kube-api-access-2n6x6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:26:12.554329 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:12.554302 2565 generic.go:356] "Generic (PLEG): container finished" podID="9a774d95-94f5-4ff3-ab69-0f32b7aba847" containerID="08aefd85713a649d11294f8d46eb95a1fe631245e259e9af13c22e3e396a3709" exitCode=0 Jul 31 19:26:12.554437 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:12.554344 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn" Jul 31 19:26:12.554437 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:12.554345 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn" event={"ID":"9a774d95-94f5-4ff3-ab69-0f32b7aba847","Type":"ContainerDied","Data":"08aefd85713a649d11294f8d46eb95a1fe631245e259e9af13c22e3e396a3709"} Jul 31 19:26:12.554531 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:12.554457 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn" event={"ID":"9a774d95-94f5-4ff3-ab69-0f32b7aba847","Type":"ContainerDied","Data":"345407ca38595ba33074a243ba75343078d503ba18891c666f987bf6743dbb9b"} Jul 31 19:26:12.554531 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:12.554477 2565 scope.go:117] "RemoveContainer" containerID="08aefd85713a649d11294f8d46eb95a1fe631245e259e9af13c22e3e396a3709" Jul 31 19:26:12.565165 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:12.565150 2565 scope.go:117] "RemoveContainer" containerID="08aefd85713a649d11294f8d46eb95a1fe631245e259e9af13c22e3e396a3709" Jul 31 19:26:12.565439 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:26:12.565418 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08aefd85713a649d11294f8d46eb95a1fe631245e259e9af13c22e3e396a3709\": container with ID starting with 08aefd85713a649d11294f8d46eb95a1fe631245e259e9af13c22e3e396a3709 not found: ID does not exist" containerID="08aefd85713a649d11294f8d46eb95a1fe631245e259e9af13c22e3e396a3709" Jul 31 19:26:12.565497 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:12.565446 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08aefd85713a649d11294f8d46eb95a1fe631245e259e9af13c22e3e396a3709"} err="failed to get container status \"08aefd85713a649d11294f8d46eb95a1fe631245e259e9af13c22e3e396a3709\": rpc error: code = NotFound desc = could not find container \"08aefd85713a649d11294f8d46eb95a1fe631245e259e9af13c22e3e396a3709\": container with ID starting with 08aefd85713a649d11294f8d46eb95a1fe631245e259e9af13c22e3e396a3709 not found: ID does not exist" Jul 31 19:26:12.588777 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:12.588756 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn"] Jul 31 19:26:12.593916 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:12.593897 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-66ce83/test-model-66ce83-kserve-844478dcff-s9tcn"] Jul 31 19:26:12.644379 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:12.644347 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a774d95-94f5-4ff3-ab69-0f32b7aba847-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:26:12.644379 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:12.644373 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n6x6\" (UniqueName: \"kubernetes.io/projected/9a774d95-94f5-4ff3-ab69-0f32b7aba847-kube-api-access-2n6x6\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:26:12.644546 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:12.644383 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a774d95-94f5-4ff3-ab69-0f32b7aba847-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:26:12.644546 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:12.644395 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a774d95-94f5-4ff3-ab69-0f32b7aba847-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:26:13.886318 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:13.886283 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a774d95-94f5-4ff3-ab69-0f32b7aba847" path="/var/lib/kubelet/pods/9a774d95-94f5-4ff3-ab69-0f32b7aba847/volumes" Jul 31 19:26:16.056078 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:16.056027 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:26:16.158109 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:16.158069 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:26:16.284681 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:16.284648 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l"] Jul 31 19:26:16.284958 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:16.284926 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" podUID="050f3f0b-8496-4f94-a881-19d1598ab6cc" containerName="istio-proxy" containerID="cri-o://21e45f105d1b40e4ce76cfcaa97d7fd58581bc4026c95c5c79bd895172d93250" gracePeriod=30 Jul 31 19:26:16.306166 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:16.306095 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:26:16.688883 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:16.688794 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:26:21.549780 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.549717 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:26:21.608638 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.608590 2565 generic.go:356] "Generic (PLEG): container finished" podID="050f3f0b-8496-4f94-a881-19d1598ab6cc" containerID="21e45f105d1b40e4ce76cfcaa97d7fd58581bc4026c95c5c79bd895172d93250" exitCode=0 Jul 31 19:26:21.608892 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.608726 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" Jul 31 19:26:21.608892 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.608746 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" event={"ID":"050f3f0b-8496-4f94-a881-19d1598ab6cc","Type":"ContainerDied","Data":"21e45f105d1b40e4ce76cfcaa97d7fd58581bc4026c95c5c79bd895172d93250"} Jul 31 19:26:21.608892 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.608838 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l" event={"ID":"050f3f0b-8496-4f94-a881-19d1598ab6cc","Type":"ContainerDied","Data":"e350afa354ebb1029621e15b96af5510a485db9bfdec2d093c3de13442e5c2fb"} Jul 31 19:26:21.608892 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.608870 2565 scope.go:117] "RemoveContainer" containerID="21e45f105d1b40e4ce76cfcaa97d7fd58581bc4026c95c5c79bd895172d93250" Jul 31 19:26:21.618459 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.618444 2565 scope.go:117] "RemoveContainer" containerID="850f417d2208de9116c14ca4a5927d5a424607848294199d1f7ee43c1bdd4d62" Jul 31 19:26:21.626706 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.626689 2565 scope.go:117] "RemoveContainer" containerID="21e45f105d1b40e4ce76cfcaa97d7fd58581bc4026c95c5c79bd895172d93250" Jul 31 19:26:21.626966 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:26:21.626943 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21e45f105d1b40e4ce76cfcaa97d7fd58581bc4026c95c5c79bd895172d93250\": container with ID starting with 21e45f105d1b40e4ce76cfcaa97d7fd58581bc4026c95c5c79bd895172d93250 not found: ID does not exist" containerID="21e45f105d1b40e4ce76cfcaa97d7fd58581bc4026c95c5c79bd895172d93250" Jul 31 19:26:21.627043 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.626973 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e45f105d1b40e4ce76cfcaa97d7fd58581bc4026c95c5c79bd895172d93250"} err="failed to get container status \"21e45f105d1b40e4ce76cfcaa97d7fd58581bc4026c95c5c79bd895172d93250\": rpc error: code = NotFound desc = could not find container \"21e45f105d1b40e4ce76cfcaa97d7fd58581bc4026c95c5c79bd895172d93250\": container with ID starting with 21e45f105d1b40e4ce76cfcaa97d7fd58581bc4026c95c5c79bd895172d93250 not found: ID does not exist" Jul 31 19:26:21.627043 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.626990 2565 scope.go:117] "RemoveContainer" containerID="850f417d2208de9116c14ca4a5927d5a424607848294199d1f7ee43c1bdd4d62" Jul 31 19:26:21.627218 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:26:21.627203 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"850f417d2208de9116c14ca4a5927d5a424607848294199d1f7ee43c1bdd4d62\": container with ID starting with 850f417d2208de9116c14ca4a5927d5a424607848294199d1f7ee43c1bdd4d62 not found: ID does not exist" containerID="850f417d2208de9116c14ca4a5927d5a424607848294199d1f7ee43c1bdd4d62" Jul 31 19:26:21.627275 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.627222 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"850f417d2208de9116c14ca4a5927d5a424607848294199d1f7ee43c1bdd4d62"} err="failed to get container status \"850f417d2208de9116c14ca4a5927d5a424607848294199d1f7ee43c1bdd4d62\": rpc error: code = NotFound desc = could not find container \"850f417d2208de9116c14ca4a5927d5a424607848294199d1f7ee43c1bdd4d62\": container with ID starting with 850f417d2208de9116c14ca4a5927d5a424607848294199d1f7ee43c1bdd4d62 not found: ID does not exist" Jul 31 19:26:21.730967 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.730938 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q69gl\" (UniqueName: \"kubernetes.io/projected/050f3f0b-8496-4f94-a881-19d1598ab6cc-kube-api-access-q69gl\") pod \"050f3f0b-8496-4f94-a881-19d1598ab6cc\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " Jul 31 19:26:21.731125 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.731034 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/050f3f0b-8496-4f94-a881-19d1598ab6cc-istio-data\") pod \"050f3f0b-8496-4f94-a881-19d1598ab6cc\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " Jul 31 19:26:21.731125 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.731061 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/050f3f0b-8496-4f94-a881-19d1598ab6cc-istio-envoy\") pod \"050f3f0b-8496-4f94-a881-19d1598ab6cc\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " Jul 31 19:26:21.731125 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.731080 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/050f3f0b-8496-4f94-a881-19d1598ab6cc-credential-socket\") pod \"050f3f0b-8496-4f94-a881-19d1598ab6cc\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " Jul 31 19:26:21.731314 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.731133 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/050f3f0b-8496-4f94-a881-19d1598ab6cc-istiod-ca-cert\") pod \"050f3f0b-8496-4f94-a881-19d1598ab6cc\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " Jul 31 19:26:21.731314 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.731163 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/050f3f0b-8496-4f94-a881-19d1598ab6cc-workload-socket\") pod \"050f3f0b-8496-4f94-a881-19d1598ab6cc\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " Jul 31 19:26:21.731314 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.731189 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/050f3f0b-8496-4f94-a881-19d1598ab6cc-workload-certs\") pod \"050f3f0b-8496-4f94-a881-19d1598ab6cc\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " Jul 31 19:26:21.731314 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.731234 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/050f3f0b-8496-4f94-a881-19d1598ab6cc-istio-podinfo\") pod \"050f3f0b-8496-4f94-a881-19d1598ab6cc\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " Jul 31 19:26:21.731314 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.731287 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/050f3f0b-8496-4f94-a881-19d1598ab6cc-istio-token\") pod \"050f3f0b-8496-4f94-a881-19d1598ab6cc\" (UID: \"050f3f0b-8496-4f94-a881-19d1598ab6cc\") " Jul 31 19:26:21.731567 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.731392 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050f3f0b-8496-4f94-a881-19d1598ab6cc-istio-data" (OuterVolumeSpecName: "istio-data") pod "050f3f0b-8496-4f94-a881-19d1598ab6cc" (UID: "050f3f0b-8496-4f94-a881-19d1598ab6cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:26:21.731567 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.731412 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050f3f0b-8496-4f94-a881-19d1598ab6cc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "050f3f0b-8496-4f94-a881-19d1598ab6cc" (UID: "050f3f0b-8496-4f94-a881-19d1598ab6cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:26:21.731668 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.731644 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050f3f0b-8496-4f94-a881-19d1598ab6cc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "050f3f0b-8496-4f94-a881-19d1598ab6cc" (UID: "050f3f0b-8496-4f94-a881-19d1598ab6cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:26:21.731712 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.731656 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/050f3f0b-8496-4f94-a881-19d1598ab6cc-credential-socket\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:26:21.731712 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.731694 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/050f3f0b-8496-4f94-a881-19d1598ab6cc-istio-data\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:26:21.731798 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.731738 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/050f3f0b-8496-4f94-a881-19d1598ab6cc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "050f3f0b-8496-4f94-a881-19d1598ab6cc" (UID: "050f3f0b-8496-4f94-a881-19d1598ab6cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 19:26:21.731835 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.731804 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050f3f0b-8496-4f94-a881-19d1598ab6cc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "050f3f0b-8496-4f94-a881-19d1598ab6cc" (UID: "050f3f0b-8496-4f94-a881-19d1598ab6cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:26:21.733274 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.733208 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050f3f0b-8496-4f94-a881-19d1598ab6cc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "050f3f0b-8496-4f94-a881-19d1598ab6cc" (UID: "050f3f0b-8496-4f94-a881-19d1598ab6cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:26:21.733274 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.733212 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/050f3f0b-8496-4f94-a881-19d1598ab6cc-kube-api-access-q69gl" (OuterVolumeSpecName: "kube-api-access-q69gl") pod "050f3f0b-8496-4f94-a881-19d1598ab6cc" (UID: "050f3f0b-8496-4f94-a881-19d1598ab6cc"). InnerVolumeSpecName "kube-api-access-q69gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:26:21.733785 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.733763 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/050f3f0b-8496-4f94-a881-19d1598ab6cc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "050f3f0b-8496-4f94-a881-19d1598ab6cc" (UID: "050f3f0b-8496-4f94-a881-19d1598ab6cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 19:26:21.733785 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.733775 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/050f3f0b-8496-4f94-a881-19d1598ab6cc-istio-token" (OuterVolumeSpecName: "istio-token") pod "050f3f0b-8496-4f94-a881-19d1598ab6cc" (UID: "050f3f0b-8496-4f94-a881-19d1598ab6cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:26:21.832912 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.832843 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/050f3f0b-8496-4f94-a881-19d1598ab6cc-istio-podinfo\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:26:21.832912 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.832867 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/050f3f0b-8496-4f94-a881-19d1598ab6cc-istio-token\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:26:21.832912 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.832877 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q69gl\" (UniqueName: \"kubernetes.io/projected/050f3f0b-8496-4f94-a881-19d1598ab6cc-kube-api-access-q69gl\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:26:21.832912 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.832887 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/050f3f0b-8496-4f94-a881-19d1598ab6cc-istio-envoy\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:26:21.832912 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.832896 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/050f3f0b-8496-4f94-a881-19d1598ab6cc-istiod-ca-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:26:21.832912 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.832904 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/050f3f0b-8496-4f94-a881-19d1598ab6cc-workload-socket\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:26:21.832912 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.832913 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/050f3f0b-8496-4f94-a881-19d1598ab6cc-workload-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:26:21.927713 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.927682 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l"] Jul 31 19:26:21.932725 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:21.932699 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-66ce83-openshift-default-85c4964bf6-tcz9l"] Jul 31 19:26:23.887318 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:23.887283 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="050f3f0b-8496-4f94-a881-19d1598ab6cc" path="/var/lib/kubelet/pods/050f3f0b-8496-4f94-a881-19d1598ab6cc/volumes" Jul 31 19:26:27.420136 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:27.420101 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:26:27.518913 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:27.518876 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:26:27.599760 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:27.599723 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:26:27.658781 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:27.658751 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:26:27.972557 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:27.972517 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:26:29.451467 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:29.451436 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:26:34.212253 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:34.212210 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:26:34.313448 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:34.313415 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:26:34.453185 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:34.453149 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7"] Jul 31 19:26:34.453413 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:34.453388 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" podUID="112c8c10-517d-4069-a825-8783c6d683a7" containerName="istio-proxy" containerID="cri-o://0a73315d8fe3a176f7bc84e0997c40ca68900b2b618565ea97f8adc73fed7f78" gracePeriod=30 Jul 31 19:26:34.480697 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:34.480605 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:26:34.697943 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:34.697908 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:26:35.084312 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.084276 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:26:35.330733 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.330697 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9"] Jul 31 19:26:35.331284 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.331225 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="050f3f0b-8496-4f94-a881-19d1598ab6cc" containerName="istio-proxy" Jul 31 19:26:35.331348 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.331287 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="050f3f0b-8496-4f94-a881-19d1598ab6cc" containerName="istio-proxy" Jul 31 19:26:35.331348 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.331326 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="050f3f0b-8496-4f94-a881-19d1598ab6cc" containerName="istio-proxy" Jul 31 19:26:35.331348 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.331333 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="050f3f0b-8496-4f94-a881-19d1598ab6cc" containerName="istio-proxy" Jul 31 19:26:35.331348 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.331347 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a774d95-94f5-4ff3-ab69-0f32b7aba847" containerName="main" Jul 31 19:26:35.331492 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.331354 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a774d95-94f5-4ff3-ab69-0f32b7aba847" containerName="main" Jul 31 19:26:35.331492 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.331430 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a774d95-94f5-4ff3-ab69-0f32b7aba847" containerName="main" Jul 31 19:26:35.331492 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.331439 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="050f3f0b-8496-4f94-a881-19d1598ab6cc" containerName="istio-proxy" Jul 31 19:26:35.331492 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.331446 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="050f3f0b-8496-4f94-a881-19d1598ab6cc" containerName="istio-proxy" Jul 31 19:26:35.333723 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.333701 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:35.335940 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.335890 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-968afa-openshift-default-dockercfg-5wndl\"" Jul 31 19:26:35.344716 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.344695 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9"] Jul 31 19:26:35.361783 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.361750 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/976f6576-26f5-466a-ac26-a861489143f5-workload-certs\") pod \"e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:35.361999 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.361893 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/976f6576-26f5-466a-ac26-a861489143f5-istio-token\") pod \"e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:35.361999 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.361933 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/976f6576-26f5-466a-ac26-a861489143f5-credential-socket\") pod \"e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:35.361999 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.361958 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/976f6576-26f5-466a-ac26-a861489143f5-workload-socket\") pod \"e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:35.362212 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.362003 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/976f6576-26f5-466a-ac26-a861489143f5-istio-podinfo\") pod \"e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:35.362212 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.362030 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/976f6576-26f5-466a-ac26-a861489143f5-istiod-ca-cert\") pod \"e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:35.362212 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.362057 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v42pf\" (UniqueName: \"kubernetes.io/projected/976f6576-26f5-466a-ac26-a861489143f5-kube-api-access-v42pf\") pod \"e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:35.362212 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.362170 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/976f6576-26f5-466a-ac26-a861489143f5-istio-data\") pod \"e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:35.362396 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.362223 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/976f6576-26f5-466a-ac26-a861489143f5-istio-envoy\") pod \"e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:35.372552 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.372525 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:26:35.463169 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.463137 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/976f6576-26f5-466a-ac26-a861489143f5-workload-certs\") pod \"e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:35.463359 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.463207 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/976f6576-26f5-466a-ac26-a861489143f5-istio-token\") pod \"e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:35.463359 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.463226 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/976f6576-26f5-466a-ac26-a861489143f5-credential-socket\") pod \"e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:35.463359 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.463275 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/976f6576-26f5-466a-ac26-a861489143f5-workload-socket\") pod \"e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:35.463359 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.463308 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/976f6576-26f5-466a-ac26-a861489143f5-istio-podinfo\") pod \"e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:35.463359 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.463328 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/976f6576-26f5-466a-ac26-a861489143f5-istiod-ca-cert\") pod \"e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:35.463631 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.463360 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v42pf\" (UniqueName: \"kubernetes.io/projected/976f6576-26f5-466a-ac26-a861489143f5-kube-api-access-v42pf\") pod \"e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:35.463631 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.463392 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/976f6576-26f5-466a-ac26-a861489143f5-istio-data\") pod \"e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:35.463631 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.463420 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/976f6576-26f5-466a-ac26-a861489143f5-istio-envoy\") pod \"e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:35.463765 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.463627 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/976f6576-26f5-466a-ac26-a861489143f5-workload-certs\") pod \"e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:35.463765 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.463656 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/976f6576-26f5-466a-ac26-a861489143f5-credential-socket\") pod \"e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:35.463765 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.463698 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/976f6576-26f5-466a-ac26-a861489143f5-workload-socket\") pod \"e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:35.463978 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.463959 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/976f6576-26f5-466a-ac26-a861489143f5-istio-data\") pod \"e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:35.464043 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.464016 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/976f6576-26f5-466a-ac26-a861489143f5-istiod-ca-cert\") pod \"e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:35.465676 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.465659 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/976f6576-26f5-466a-ac26-a861489143f5-istio-envoy\") pod \"e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:35.465747 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.465732 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/976f6576-26f5-466a-ac26-a861489143f5-istio-podinfo\") pod \"e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:35.474988 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.474965 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/976f6576-26f5-466a-ac26-a861489143f5-istio-token\") pod \"e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:35.475339 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.475317 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v42pf\" (UniqueName: \"kubernetes.io/projected/976f6576-26f5-466a-ac26-a861489143f5-kube-api-access-v42pf\") pod \"e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:35.523015 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.522982 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:26:35.645069 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.644986 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:35.786226 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.786202 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9"] Jul 31 19:26:35.788675 ip-10-0-139-39 kubenswrapper[2565]: W0731 19:26:35.788650 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod976f6576_26f5_466a_ac26_a861489143f5.slice/crio-ffacd93d2ef174eb8e5c5e337318f0ef4976b19e2f2020ea566939e9207f5408 WatchSource:0}: Error finding container ffacd93d2ef174eb8e5c5e337318f0ef4976b19e2f2020ea566939e9207f5408: Status 404 returned error can't find the container with id ffacd93d2ef174eb8e5c5e337318f0ef4976b19e2f2020ea566939e9207f5408 Jul 31 19:26:35.790902 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.790871 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:26:35.790967 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.790936 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:26:35.791006 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:35.790980 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:26:36.692097 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:36.692063 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" event={"ID":"976f6576-26f5-466a-ac26-a861489143f5","Type":"ContainerStarted","Data":"3490f69fa6846c8092d6bf64b9ee7519ae9103449fc62f4295a82009c27d8aba"} Jul 31 19:26:36.692505 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:36.692113 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" event={"ID":"976f6576-26f5-466a-ac26-a861489143f5","Type":"ContainerStarted","Data":"ffacd93d2ef174eb8e5c5e337318f0ef4976b19e2f2020ea566939e9207f5408"} Jul 31 19:26:36.715375 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:36.715307 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" podStartSLOduration=1.715288232 podStartE2EDuration="1.715288232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:26:36.712405791 +0000 UTC m=+2573.350395624" watchObservedRunningTime="2026-07-31 19:26:36.715288232 +0000 UTC m=+2573.353278033" Jul 31 19:26:37.645682 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:37.645645 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:37.650627 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:37.650599 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:37.697926 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:37.697896 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:37.698956 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:37.698933 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:38.592965 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:38.592930 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:26:38.631831 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:38.631800 2565 patch_prober.go:28] interesting pod/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 31 19:26:38.631983 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:38.631853 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" podUID="112c8c10-517d-4069-a825-8783c6d683a7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 31 19:26:38.693352 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:38.693314 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:26:39.710541 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.710513 2565 generic.go:356] "Generic (PLEG): container finished" podID="112c8c10-517d-4069-a825-8783c6d683a7" containerID="0a73315d8fe3a176f7bc84e0997c40ca68900b2b618565ea97f8adc73fed7f78" exitCode=0 Jul 31 19:26:39.710952 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.710586 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" event={"ID":"112c8c10-517d-4069-a825-8783c6d683a7","Type":"ContainerDied","Data":"0a73315d8fe3a176f7bc84e0997c40ca68900b2b618565ea97f8adc73fed7f78"} Jul 31 19:26:39.710952 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.710639 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" event={"ID":"112c8c10-517d-4069-a825-8783c6d683a7","Type":"ContainerDied","Data":"8da1fe46bfcb978b86dcec2ebccaadeff223ddf5f2516647c76f55a3bddd8b6a"} Jul 31 19:26:39.710952 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.710664 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8da1fe46bfcb978b86dcec2ebccaadeff223ddf5f2516647c76f55a3bddd8b6a" Jul 31 19:26:39.710952 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.710683 2565 scope.go:117] "RemoveContainer" containerID="8ea65fc3fd37e1ee82120d832b4055f216d8510e03cec387726d48ece94e0bab" Jul 31 19:26:39.727481 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.727449 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:26:39.728472 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.728456 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:26:39.802532 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.802502 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/112c8c10-517d-4069-a825-8783c6d683a7-istio-podinfo\") pod \"112c8c10-517d-4069-a825-8783c6d683a7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " Jul 31 19:26:39.802710 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.802554 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/112c8c10-517d-4069-a825-8783c6d683a7-workload-certs\") pod \"112c8c10-517d-4069-a825-8783c6d683a7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " Jul 31 19:26:39.802710 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.802592 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/112c8c10-517d-4069-a825-8783c6d683a7-credential-socket\") pod \"112c8c10-517d-4069-a825-8783c6d683a7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " Jul 31 19:26:39.802710 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.802638 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/112c8c10-517d-4069-a825-8783c6d683a7-workload-socket\") pod \"112c8c10-517d-4069-a825-8783c6d683a7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " Jul 31 19:26:39.802710 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.802666 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x89js\" (UniqueName: \"kubernetes.io/projected/112c8c10-517d-4069-a825-8783c6d683a7-kube-api-access-x89js\") pod \"112c8c10-517d-4069-a825-8783c6d683a7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " Jul 31 19:26:39.802710 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.802700 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/112c8c10-517d-4069-a825-8783c6d683a7-istiod-ca-cert\") pod \"112c8c10-517d-4069-a825-8783c6d683a7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " Jul 31 19:26:39.802980 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.802767 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/112c8c10-517d-4069-a825-8783c6d683a7-istio-data\") pod \"112c8c10-517d-4069-a825-8783c6d683a7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " Jul 31 19:26:39.802980 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.802797 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/112c8c10-517d-4069-a825-8783c6d683a7-istio-token\") pod \"112c8c10-517d-4069-a825-8783c6d683a7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " Jul 31 19:26:39.802980 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.802824 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/112c8c10-517d-4069-a825-8783c6d683a7-istio-envoy\") pod \"112c8c10-517d-4069-a825-8783c6d683a7\" (UID: \"112c8c10-517d-4069-a825-8783c6d683a7\") " Jul 31 19:26:39.802980 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.802933 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/112c8c10-517d-4069-a825-8783c6d683a7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "112c8c10-517d-4069-a825-8783c6d683a7" (UID: "112c8c10-517d-4069-a825-8783c6d683a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:26:39.802980 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.802920 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/112c8c10-517d-4069-a825-8783c6d683a7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "112c8c10-517d-4069-a825-8783c6d683a7" (UID: "112c8c10-517d-4069-a825-8783c6d683a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:26:39.803258 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.802983 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/112c8c10-517d-4069-a825-8783c6d683a7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "112c8c10-517d-4069-a825-8783c6d683a7" (UID: "112c8c10-517d-4069-a825-8783c6d683a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:26:39.803258 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.803162 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/112c8c10-517d-4069-a825-8783c6d683a7-istio-data" (OuterVolumeSpecName: "istio-data") pod "112c8c10-517d-4069-a825-8783c6d683a7" (UID: "112c8c10-517d-4069-a825-8783c6d683a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:26:39.803258 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.803197 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/112c8c10-517d-4069-a825-8783c6d683a7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "112c8c10-517d-4069-a825-8783c6d683a7" (UID: "112c8c10-517d-4069-a825-8783c6d683a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 19:26:39.803447 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.803360 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/112c8c10-517d-4069-a825-8783c6d683a7-workload-socket\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:26:39.803447 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.803384 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/112c8c10-517d-4069-a825-8783c6d683a7-istiod-ca-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:26:39.803447 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.803397 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/112c8c10-517d-4069-a825-8783c6d683a7-istio-data\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:26:39.803447 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.803409 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/112c8c10-517d-4069-a825-8783c6d683a7-workload-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:26:39.803447 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.803421 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/112c8c10-517d-4069-a825-8783c6d683a7-credential-socket\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:26:39.804881 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.804850 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/112c8c10-517d-4069-a825-8783c6d683a7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "112c8c10-517d-4069-a825-8783c6d683a7" (UID: "112c8c10-517d-4069-a825-8783c6d683a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 19:26:39.804979 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.804891 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/112c8c10-517d-4069-a825-8783c6d683a7-kube-api-access-x89js" (OuterVolumeSpecName: "kube-api-access-x89js") pod "112c8c10-517d-4069-a825-8783c6d683a7" (UID: "112c8c10-517d-4069-a825-8783c6d683a7"). InnerVolumeSpecName "kube-api-access-x89js". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:26:39.805099 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.805081 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/112c8c10-517d-4069-a825-8783c6d683a7-istio-token" (OuterVolumeSpecName: "istio-token") pod "112c8c10-517d-4069-a825-8783c6d683a7" (UID: "112c8c10-517d-4069-a825-8783c6d683a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:26:39.805145 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.805107 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/112c8c10-517d-4069-a825-8783c6d683a7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "112c8c10-517d-4069-a825-8783c6d683a7" (UID: "112c8c10-517d-4069-a825-8783c6d683a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:26:39.904967 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.904938 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x89js\" (UniqueName: \"kubernetes.io/projected/112c8c10-517d-4069-a825-8783c6d683a7-kube-api-access-x89js\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:26:39.904967 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.904968 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/112c8c10-517d-4069-a825-8783c6d683a7-istio-token\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:26:39.905160 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.904982 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/112c8c10-517d-4069-a825-8783c6d683a7-istio-envoy\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:26:39.905160 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:39.904990 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/112c8c10-517d-4069-a825-8783c6d683a7-istio-podinfo\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:26:40.038438 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:40.038405 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:26:40.271432 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:40.271397 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:26:40.399504 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:40.399419 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:26:40.716909 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:40.716833 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7" Jul 31 19:26:40.750651 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:40.750620 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7"] Jul 31 19:26:40.754932 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:40.754910 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8aa450-openshift-default-864bb56cf8-n89n7"] Jul 31 19:26:41.887163 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:41.887129 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="112c8c10-517d-4069-a825-8783c6d683a7" path="/var/lib/kubelet/pods/112c8c10-517d-4069-a825-8783c6d683a7/volumes" Jul 31 19:26:42.467190 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:42.467152 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:26:52.700016 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:52.699976 2565 patch_prober.go:28] interesting pod/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 31 19:26:52.700474 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:52.700081 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" podUID="976f6576-26f5-466a-ac26-a861489143f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 31 19:26:52.787303 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:52.787270 2565 generic.go:356] "Generic (PLEG): container finished" podID="976f6576-26f5-466a-ac26-a861489143f5" containerID="3490f69fa6846c8092d6bf64b9ee7519ae9103449fc62f4295a82009c27d8aba" exitCode=0 Jul 31 19:26:52.787475 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:52.787342 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" event={"ID":"976f6576-26f5-466a-ac26-a861489143f5","Type":"ContainerDied","Data":"3490f69fa6846c8092d6bf64b9ee7519ae9103449fc62f4295a82009c27d8aba"} Jul 31 19:26:52.787853 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:52.787834 2565 scope.go:117] "RemoveContainer" containerID="3490f69fa6846c8092d6bf64b9ee7519ae9103449fc62f4295a82009c27d8aba" Jul 31 19:26:52.788999 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:52.788965 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:26:52.789077 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:52.789061 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:26:52.789117 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:52.789101 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:26:53.794206 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:53.794172 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" event={"ID":"976f6576-26f5-466a-ac26-a861489143f5","Type":"ContainerStarted","Data":"1186995b43a52446c0d3797f790f5dabc0519ec2d49f05c94030206839b69932"} Jul 31 19:26:54.646410 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:54.646373 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:54.646818 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:54.646728 2565 patch_prober.go:28] interesting pod/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 31 19:26:54.647004 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:54.646980 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" podUID="976f6576-26f5-466a-ac26-a861489143f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 31 19:26:55.645819 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:55.645781 2565 patch_prober.go:28] interesting pod/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 31 19:26:55.646443 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:55.646407 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" podUID="976f6576-26f5-466a-ac26-a861489143f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 31 19:26:56.645518 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:56.645474 2565 patch_prober.go:28] interesting pod/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 31 19:26:56.645733 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:56.645566 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" podUID="976f6576-26f5-466a-ac26-a861489143f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 31 19:26:57.646393 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:57.646353 2565 patch_prober.go:28] interesting pod/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 31 19:26:57.647025 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:57.646983 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" podUID="976f6576-26f5-466a-ac26-a861489143f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 31 19:26:58.646322 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:58.646278 2565 patch_prober.go:28] interesting pod/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 31 19:26:58.646541 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:58.646348 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" podUID="976f6576-26f5-466a-ac26-a861489143f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 31 19:26:59.649536 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:59.649506 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:59.649901 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:59.649767 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:26:59.650627 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:26:59.650610 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:27:03.549081 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:03.549041 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:27:03.996571 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:03.996537 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:27:04.271986 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.271867 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9"] Jul 31 19:27:04.272486 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.272470 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="112c8c10-517d-4069-a825-8783c6d683a7" containerName="istio-proxy" Jul 31 19:27:04.272557 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.272491 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="112c8c10-517d-4069-a825-8783c6d683a7" containerName="istio-proxy" Jul 31 19:27:04.272557 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.272512 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="112c8c10-517d-4069-a825-8783c6d683a7" containerName="istio-proxy" Jul 31 19:27:04.272557 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.272520 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="112c8c10-517d-4069-a825-8783c6d683a7" containerName="istio-proxy" Jul 31 19:27:04.272661 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.272604 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="112c8c10-517d-4069-a825-8783c6d683a7" containerName="istio-proxy" Jul 31 19:27:04.274958 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.274936 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:04.276790 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.276767 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f3dafe-openshift-default-dockercfg-7wtqv\"" Jul 31 19:27:04.287215 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.287187 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9"] Jul 31 19:27:04.300041 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.300014 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:27:04.347719 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.347681 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7085d14a-94c1-4c57-a4d5-b1169813eb77-istiod-ca-cert\") pod \"e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:04.347893 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.347730 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89lcw\" (UniqueName: \"kubernetes.io/projected/7085d14a-94c1-4c57-a4d5-b1169813eb77-kube-api-access-89lcw\") pod \"e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:04.347893 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.347778 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7085d14a-94c1-4c57-a4d5-b1169813eb77-istio-token\") pod \"e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:04.347893 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.347803 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7085d14a-94c1-4c57-a4d5-b1169813eb77-istio-envoy\") pod \"e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:04.347893 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.347870 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7085d14a-94c1-4c57-a4d5-b1169813eb77-istio-data\") pod \"e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:04.348089 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.347913 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7085d14a-94c1-4c57-a4d5-b1169813eb77-workload-socket\") pod \"e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:04.348089 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.347944 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7085d14a-94c1-4c57-a4d5-b1169813eb77-workload-certs\") pod \"e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:04.348089 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.347985 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7085d14a-94c1-4c57-a4d5-b1169813eb77-credential-socket\") pod \"e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:04.348089 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.348018 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7085d14a-94c1-4c57-a4d5-b1169813eb77-istio-podinfo\") pod \"e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:04.448661 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.448614 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7085d14a-94c1-4c57-a4d5-b1169813eb77-istiod-ca-cert\") pod \"e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:04.448661 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.448671 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89lcw\" (UniqueName: \"kubernetes.io/projected/7085d14a-94c1-4c57-a4d5-b1169813eb77-kube-api-access-89lcw\") pod \"e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:04.448908 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.448695 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7085d14a-94c1-4c57-a4d5-b1169813eb77-istio-token\") pod \"e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:04.448908 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.448712 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7085d14a-94c1-4c57-a4d5-b1169813eb77-istio-envoy\") pod \"e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:04.448908 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.448739 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7085d14a-94c1-4c57-a4d5-b1169813eb77-istio-data\") pod \"e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:04.448908 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.448779 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7085d14a-94c1-4c57-a4d5-b1169813eb77-workload-socket\") pod \"e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:04.448908 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.448824 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7085d14a-94c1-4c57-a4d5-b1169813eb77-workload-certs\") pod \"e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:04.449162 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.448962 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7085d14a-94c1-4c57-a4d5-b1169813eb77-credential-socket\") pod \"e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:04.449162 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.449005 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7085d14a-94c1-4c57-a4d5-b1169813eb77-istio-podinfo\") pod \"e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:04.449162 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.449145 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7085d14a-94c1-4c57-a4d5-b1169813eb77-istio-data\") pod \"e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:04.449349 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.449199 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7085d14a-94c1-4c57-a4d5-b1169813eb77-workload-certs\") pod \"e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:04.449411 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.449349 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7085d14a-94c1-4c57-a4d5-b1169813eb77-workload-socket\") pod \"e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:04.449526 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.449502 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7085d14a-94c1-4c57-a4d5-b1169813eb77-istiod-ca-cert\") pod \"e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:04.449651 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.449603 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7085d14a-94c1-4c57-a4d5-b1169813eb77-credential-socket\") pod \"e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:04.451093 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.451074 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7085d14a-94c1-4c57-a4d5-b1169813eb77-istio-envoy\") pod \"e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:04.451420 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.451404 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7085d14a-94c1-4c57-a4d5-b1169813eb77-istio-podinfo\") pod \"e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:04.459982 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.459954 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7085d14a-94c1-4c57-a4d5-b1169813eb77-istio-token\") pod \"e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:04.460515 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.460497 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89lcw\" (UniqueName: \"kubernetes.io/projected/7085d14a-94c1-4c57-a4d5-b1169813eb77-kube-api-access-89lcw\") pod \"e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:04.585594 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.585507 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:04.684794 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.684749 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:27:04.741571 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.741540 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9"] Jul 31 19:27:04.744726 ip-10-0-139-39 kubenswrapper[2565]: W0731 19:27:04.744691 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7085d14a_94c1_4c57_a4d5_b1169813eb77.slice/crio-1cb76ef03b7c2cf66d993dff7b071fbbe50f9463454306e503ce6cc2c98706c2 WatchSource:0}: Error finding container 1cb76ef03b7c2cf66d993dff7b071fbbe50f9463454306e503ce6cc2c98706c2: Status 404 returned error can't find the container with id 1cb76ef03b7c2cf66d993dff7b071fbbe50f9463454306e503ce6cc2c98706c2 Jul 31 19:27:04.747078 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.747036 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:27:04.747163 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.747128 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:27:04.747199 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.747173 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:27:04.850915 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.850817 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" event={"ID":"7085d14a-94c1-4c57-a4d5-b1169813eb77","Type":"ContainerStarted","Data":"ddc519c7f80f576cd45d5449fc4cb9582a34e325a58d88951abbafe30ccfc985"} Jul 31 19:27:04.850915 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.850883 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" event={"ID":"7085d14a-94c1-4c57-a4d5-b1169813eb77","Type":"ContainerStarted","Data":"1cb76ef03b7c2cf66d993dff7b071fbbe50f9463454306e503ce6cc2c98706c2"} Jul 31 19:27:04.873785 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:04.873726 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" podStartSLOduration=0.873710007 podStartE2EDuration="873.710007ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:27:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:27:04.871868191 +0000 UTC m=+2601.509857991" watchObservedRunningTime="2026-07-31 19:27:04.873710007 +0000 UTC m=+2601.511699807" Jul 31 19:27:05.586168 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:05.586128 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:05.591289 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:05.591265 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:05.856823 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:05.856729 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:05.857714 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:05.857692 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:06.817790 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:06.817703 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:27:06.916093 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:06.916046 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:27:08.104086 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:08.104031 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:27:08.464744 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:08.464645 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:27:08.619353 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:08.619301 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:27:08.731989 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:08.731882 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:27:10.859617 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:10.859582 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:27:20.858951 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:20.858911 2565 patch_prober.go:28] interesting pod/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 31 19:27:20.859413 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:20.858988 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" podUID="7085d14a-94c1-4c57-a4d5-b1169813eb77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 31 19:27:20.939787 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:20.939710 2565 generic.go:356] "Generic (PLEG): container finished" podID="7085d14a-94c1-4c57-a4d5-b1169813eb77" containerID="ddc519c7f80f576cd45d5449fc4cb9582a34e325a58d88951abbafe30ccfc985" exitCode=0 Jul 31 19:27:20.939787 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:20.939772 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" event={"ID":"7085d14a-94c1-4c57-a4d5-b1169813eb77","Type":"ContainerDied","Data":"ddc519c7f80f576cd45d5449fc4cb9582a34e325a58d88951abbafe30ccfc985"} Jul 31 19:27:20.940213 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:20.940194 2565 scope.go:117] "RemoveContainer" containerID="ddc519c7f80f576cd45d5449fc4cb9582a34e325a58d88951abbafe30ccfc985" Jul 31 19:27:20.941275 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:20.941228 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:27:20.941375 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:20.941355 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:27:20.941412 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:20.941398 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:27:21.950744 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:21.950688 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" event={"ID":"7085d14a-94c1-4c57-a4d5-b1169813eb77","Type":"ContainerStarted","Data":"3876a2a0d88e0222c809ec4aa79b49a37b10bb53bdb06c92e3ebf5bfac02d626"} Jul 31 19:27:22.586316 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:22.586281 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:22.586561 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:22.586537 2565 patch_prober.go:28] interesting pod/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 31 19:27:22.586613 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:22.586586 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" podUID="7085d14a-94c1-4c57-a4d5-b1169813eb77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 31 19:27:23.586508 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:23.586467 2565 patch_prober.go:28] interesting pod/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 31 19:27:23.586986 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:23.586530 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" podUID="7085d14a-94c1-4c57-a4d5-b1169813eb77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 31 19:27:24.586757 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:24.586711 2565 patch_prober.go:28] interesting pod/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 31 19:27:24.587153 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:24.586778 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" podUID="7085d14a-94c1-4c57-a4d5-b1169813eb77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 31 19:27:25.586072 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:25.586031 2565 patch_prober.go:28] interesting pod/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 31 19:27:25.586280 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:25.586089 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" podUID="7085d14a-94c1-4c57-a4d5-b1169813eb77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 31 19:27:26.586459 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:26.586413 2565 patch_prober.go:28] interesting pod/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 31 19:27:26.586843 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:26.586472 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" podUID="7085d14a-94c1-4c57-a4d5-b1169813eb77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 31 19:27:27.589738 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:27.589703 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:27.590177 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:27.590055 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:27.590854 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:27.590837 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:27:31.720736 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:31.720697 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:27:34.640598 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:34.640549 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968afa-85d5f"] Jul 31 19:27:34.641224 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:34.641201 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="112c8c10-517d-4069-a825-8783c6d683a7" containerName="istio-proxy" Jul 31 19:27:34.643379 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:34.643357 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968afa-85d5f" Jul 31 19:27:34.645250 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:34.645217 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4qkh\"" Jul 31 19:27:34.645352 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:34.645222 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 31 19:27:34.651282 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:34.651255 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968afa-85d5f"] Jul 31 19:27:34.766863 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:34.766832 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28969727-0efa-4615-a4fb-29ffb44a6b78-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-968afa-85d5f\" (UID: \"28969727-0efa-4615-a4fb-29ffb44a6b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968afa-85d5f" Jul 31 19:27:34.867885 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:34.867847 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28969727-0efa-4615-a4fb-29ffb44a6b78-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-968afa-85d5f\" (UID: \"28969727-0efa-4615-a4fb-29ffb44a6b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968afa-85d5f" Jul 31 19:27:34.868538 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:34.868518 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28969727-0efa-4615-a4fb-29ffb44a6b78-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-968afa-85d5f\" (UID: \"28969727-0efa-4615-a4fb-29ffb44a6b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968afa-85d5f" Jul 31 19:27:34.955795 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:34.955713 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968afa-85d5f" Jul 31 19:27:35.297122 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:35.297097 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968afa-85d5f"] Jul 31 19:27:35.299444 ip-10-0-139-39 kubenswrapper[2565]: W0731 19:27:35.299417 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28969727_0efa_4615_a4fb_29ffb44a6b78.slice/crio-a14b90e71fc333b65737b1076295df24266884a4432d34368e7618195e55f7b7 WatchSource:0}: Error finding container a14b90e71fc333b65737b1076295df24266884a4432d34368e7618195e55f7b7: Status 404 returned error can't find the container with id a14b90e71fc333b65737b1076295df24266884a4432d34368e7618195e55f7b7 Jul 31 19:27:36.029402 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:36.029365 2565 generic.go:356] "Generic (PLEG): container finished" podID="28969727-0efa-4615-a4fb-29ffb44a6b78" containerID="1f31172d9ec76fdb0778d40358e45b06e664abd1811ed55608e5876f2bd8f453" exitCode=0 Jul 31 19:27:36.029824 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:36.029448 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968afa-85d5f" event={"ID":"28969727-0efa-4615-a4fb-29ffb44a6b78","Type":"ContainerDied","Data":"1f31172d9ec76fdb0778d40358e45b06e664abd1811ed55608e5876f2bd8f453"} Jul 31 19:27:36.029824 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:36.029497 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968afa-85d5f" event={"ID":"28969727-0efa-4615-a4fb-29ffb44a6b78","Type":"ContainerStarted","Data":"a14b90e71fc333b65737b1076295df24266884a4432d34368e7618195e55f7b7"} Jul 31 19:27:37.174518 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:37.174485 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968afa-85d5f" Jul 31 19:27:37.188171 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:37.188140 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28969727-0efa-4615-a4fb-29ffb44a6b78-maas-api-service-ca\") pod \"28969727-0efa-4615-a4fb-29ffb44a6b78\" (UID: \"28969727-0efa-4615-a4fb-29ffb44a6b78\") " Jul 31 19:27:37.188482 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:37.188461 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28969727-0efa-4615-a4fb-29ffb44a6b78-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "28969727-0efa-4615-a4fb-29ffb44a6b78" (UID: "28969727-0efa-4615-a4fb-29ffb44a6b78"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 19:27:37.289613 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:37.289534 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28969727-0efa-4615-a4fb-29ffb44a6b78-maas-api-service-ca\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:27:38.040961 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:38.040932 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968afa-85d5f" Jul 31 19:27:38.040961 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:38.040939 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968afa-85d5f" event={"ID":"28969727-0efa-4615-a4fb-29ffb44a6b78","Type":"ContainerDied","Data":"a14b90e71fc333b65737b1076295df24266884a4432d34368e7618195e55f7b7"} Jul 31 19:27:38.041213 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:38.040982 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a14b90e71fc333b65737b1076295df24266884a4432d34368e7618195e55f7b7" Jul 31 19:27:50.205565 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:50.205519 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:27:50.274903 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:50.274870 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:27:50.341390 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:50.341361 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:27:50.396974 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:50.396945 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968afa-85d5f"] Jul 31 19:27:50.400352 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:50.400327 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968afa-85d5f"] Jul 31 19:27:50.406993 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:50.406964 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:27:50.758737 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:50.758705 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:27:51.886645 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:51.886612 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28969727-0efa-4615-a4fb-29ffb44a6b78" path="/var/lib/kubelet/pods/28969727-0efa-4615-a4fb-29ffb44a6b78/volumes" Jul 31 19:27:52.759878 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:52.759834 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:27:57.050826 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:57.050790 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:27:57.158921 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:57.158885 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:27:57.292570 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:57.292537 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9"] Jul 31 19:27:57.292778 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:57.292754 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" podUID="976f6576-26f5-466a-ac26-a861489143f5" containerName="istio-proxy" containerID="cri-o://1186995b43a52446c0d3797f790f5dabc0519ec2d49f05c94030206839b69932" gracePeriod=30 Jul 31 19:27:57.318668 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:57.318583 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:27:57.537825 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:57.537794 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:27:59.652403 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:59.652363 2565 patch_prober.go:28] interesting pod/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 31 19:27:59.652858 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:27:59.652424 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" podUID="976f6576-26f5-466a-ac26-a861489143f5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 31 19:28:02.550635 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:02.550610 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:28:02.620288 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:02.620190 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/976f6576-26f5-466a-ac26-a861489143f5-istiod-ca-cert\") pod \"976f6576-26f5-466a-ac26-a861489143f5\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " Jul 31 19:28:02.620427 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:02.620310 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v42pf\" (UniqueName: \"kubernetes.io/projected/976f6576-26f5-466a-ac26-a861489143f5-kube-api-access-v42pf\") pod \"976f6576-26f5-466a-ac26-a861489143f5\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " Jul 31 19:28:02.620427 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:02.620342 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/976f6576-26f5-466a-ac26-a861489143f5-workload-certs\") pod \"976f6576-26f5-466a-ac26-a861489143f5\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " Jul 31 19:28:02.620427 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:02.620386 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/976f6576-26f5-466a-ac26-a861489143f5-credential-socket\") pod \"976f6576-26f5-466a-ac26-a861489143f5\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " Jul 31 19:28:02.620427 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:02.620421 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/976f6576-26f5-466a-ac26-a861489143f5-istio-podinfo\") pod \"976f6576-26f5-466a-ac26-a861489143f5\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " Jul 31 19:28:02.620659 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:02.620442 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/976f6576-26f5-466a-ac26-a861489143f5-istio-envoy\") pod \"976f6576-26f5-466a-ac26-a861489143f5\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " Jul 31 19:28:02.620659 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:02.620481 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/976f6576-26f5-466a-ac26-a861489143f5-istio-data\") pod \"976f6576-26f5-466a-ac26-a861489143f5\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " Jul 31 19:28:02.620659 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:02.620517 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/976f6576-26f5-466a-ac26-a861489143f5-istio-token\") pod \"976f6576-26f5-466a-ac26-a861489143f5\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " Jul 31 19:28:02.620659 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:02.620598 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/976f6576-26f5-466a-ac26-a861489143f5-workload-socket\") pod \"976f6576-26f5-466a-ac26-a861489143f5\" (UID: \"976f6576-26f5-466a-ac26-a861489143f5\") " Jul 31 19:28:02.620659 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:02.620622 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/976f6576-26f5-466a-ac26-a861489143f5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "976f6576-26f5-466a-ac26-a861489143f5" (UID: "976f6576-26f5-466a-ac26-a861489143f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 19:28:02.620940 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:02.620755 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976f6576-26f5-466a-ac26-a861489143f5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "976f6576-26f5-466a-ac26-a861489143f5" (UID: "976f6576-26f5-466a-ac26-a861489143f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:28:02.620940 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:02.620771 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976f6576-26f5-466a-ac26-a861489143f5-istio-data" (OuterVolumeSpecName: "istio-data") pod "976f6576-26f5-466a-ac26-a861489143f5" (UID: "976f6576-26f5-466a-ac26-a861489143f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:28:02.621054 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:02.620972 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976f6576-26f5-466a-ac26-a861489143f5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "976f6576-26f5-466a-ac26-a861489143f5" (UID: "976f6576-26f5-466a-ac26-a861489143f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:28:02.621054 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:02.620987 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/976f6576-26f5-466a-ac26-a861489143f5-istiod-ca-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:28:02.621054 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:02.621018 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976f6576-26f5-466a-ac26-a861489143f5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "976f6576-26f5-466a-ac26-a861489143f5" (UID: "976f6576-26f5-466a-ac26-a861489143f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:28:02.621054 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:02.621049 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/976f6576-26f5-466a-ac26-a861489143f5-workload-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:28:02.621282 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:02.621063 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/976f6576-26f5-466a-ac26-a861489143f5-istio-data\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:28:02.622601 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:02.622571 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/976f6576-26f5-466a-ac26-a861489143f5-kube-api-access-v42pf" (OuterVolumeSpecName: "kube-api-access-v42pf") pod "976f6576-26f5-466a-ac26-a861489143f5" (UID: "976f6576-26f5-466a-ac26-a861489143f5"). InnerVolumeSpecName "kube-api-access-v42pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:28:02.622739 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:02.622599 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/976f6576-26f5-466a-ac26-a861489143f5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "976f6576-26f5-466a-ac26-a861489143f5" (UID: "976f6576-26f5-466a-ac26-a861489143f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 19:28:02.622739 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:02.622617 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976f6576-26f5-466a-ac26-a861489143f5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "976f6576-26f5-466a-ac26-a861489143f5" (UID: "976f6576-26f5-466a-ac26-a861489143f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:28:02.622739 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:02.622707 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/976f6576-26f5-466a-ac26-a861489143f5-istio-token" (OuterVolumeSpecName: "istio-token") pod "976f6576-26f5-466a-ac26-a861489143f5" (UID: "976f6576-26f5-466a-ac26-a861489143f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:28:02.721920 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:02.721882 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/976f6576-26f5-466a-ac26-a861489143f5-istio-token\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:28:02.721920 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:02.721910 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/976f6576-26f5-466a-ac26-a861489143f5-workload-socket\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:28:02.721920 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:02.721920 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v42pf\" (UniqueName: \"kubernetes.io/projected/976f6576-26f5-466a-ac26-a861489143f5-kube-api-access-v42pf\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:28:02.722147 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:02.721930 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/976f6576-26f5-466a-ac26-a861489143f5-credential-socket\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:28:02.722147 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:02.721938 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/976f6576-26f5-466a-ac26-a861489143f5-istio-podinfo\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:28:02.722147 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:02.721946 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/976f6576-26f5-466a-ac26-a861489143f5-istio-envoy\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:28:03.172908 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:03.172873 2565 generic.go:356] "Generic (PLEG): container finished" podID="976f6576-26f5-466a-ac26-a861489143f5" containerID="1186995b43a52446c0d3797f790f5dabc0519ec2d49f05c94030206839b69932" exitCode=0 Jul 31 19:28:03.173099 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:03.172930 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" Jul 31 19:28:03.173099 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:03.172942 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" event={"ID":"976f6576-26f5-466a-ac26-a861489143f5","Type":"ContainerDied","Data":"1186995b43a52446c0d3797f790f5dabc0519ec2d49f05c94030206839b69932"} Jul 31 19:28:03.173099 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:03.172996 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9" event={"ID":"976f6576-26f5-466a-ac26-a861489143f5","Type":"ContainerDied","Data":"ffacd93d2ef174eb8e5c5e337318f0ef4976b19e2f2020ea566939e9207f5408"} Jul 31 19:28:03.173099 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:03.173026 2565 scope.go:117] "RemoveContainer" containerID="1186995b43a52446c0d3797f790f5dabc0519ec2d49f05c94030206839b69932" Jul 31 19:28:03.182418 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:03.182332 2565 scope.go:117] "RemoveContainer" containerID="3490f69fa6846c8092d6bf64b9ee7519ae9103449fc62f4295a82009c27d8aba" Jul 31 19:28:03.190901 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:03.190882 2565 scope.go:117] "RemoveContainer" containerID="1186995b43a52446c0d3797f790f5dabc0519ec2d49f05c94030206839b69932" Jul 31 19:28:03.191152 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:28:03.191132 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1186995b43a52446c0d3797f790f5dabc0519ec2d49f05c94030206839b69932\": container with ID starting with 1186995b43a52446c0d3797f790f5dabc0519ec2d49f05c94030206839b69932 not found: ID does not exist" containerID="1186995b43a52446c0d3797f790f5dabc0519ec2d49f05c94030206839b69932" Jul 31 19:28:03.191214 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:03.191160 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1186995b43a52446c0d3797f790f5dabc0519ec2d49f05c94030206839b69932"} err="failed to get container status \"1186995b43a52446c0d3797f790f5dabc0519ec2d49f05c94030206839b69932\": rpc error: code = NotFound desc = could not find container \"1186995b43a52446c0d3797f790f5dabc0519ec2d49f05c94030206839b69932\": container with ID starting with 1186995b43a52446c0d3797f790f5dabc0519ec2d49f05c94030206839b69932 not found: ID does not exist" Jul 31 19:28:03.191214 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:03.191177 2565 scope.go:117] "RemoveContainer" containerID="3490f69fa6846c8092d6bf64b9ee7519ae9103449fc62f4295a82009c27d8aba" Jul 31 19:28:03.191432 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:28:03.191412 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3490f69fa6846c8092d6bf64b9ee7519ae9103449fc62f4295a82009c27d8aba\": container with ID starting with 3490f69fa6846c8092d6bf64b9ee7519ae9103449fc62f4295a82009c27d8aba not found: ID does not exist" containerID="3490f69fa6846c8092d6bf64b9ee7519ae9103449fc62f4295a82009c27d8aba" Jul 31 19:28:03.191487 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:03.191444 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3490f69fa6846c8092d6bf64b9ee7519ae9103449fc62f4295a82009c27d8aba"} err="failed to get container status \"3490f69fa6846c8092d6bf64b9ee7519ae9103449fc62f4295a82009c27d8aba\": rpc error: code = NotFound desc = could not find container \"3490f69fa6846c8092d6bf64b9ee7519ae9103449fc62f4295a82009c27d8aba\": container with ID starting with 3490f69fa6846c8092d6bf64b9ee7519ae9103449fc62f4295a82009c27d8aba not found: ID does not exist" Jul 31 19:28:03.195434 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:03.195411 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9"] Jul 31 19:28:03.200408 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:03.200388 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-968afa-openshift-default-56f956d44b-xc6r9"] Jul 31 19:28:03.886823 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:03.886788 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="976f6576-26f5-466a-ac26-a861489143f5" path="/var/lib/kubelet/pods/976f6576-26f5-466a-ac26-a861489143f5/volumes" Jul 31 19:28:13.391813 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:13.391777 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:28:13.463980 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:13.463946 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:28:13.527062 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:13.527024 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:28:13.628542 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:13.628509 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:28:14.675416 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:14.675382 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:28:15.347530 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:15.347497 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:28:20.833586 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:20.833554 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:28:20.896649 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:20.896619 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:28:21.220044 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:21.219955 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9"] Jul 31 19:28:21.220585 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:21.220548 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" podUID="7085d14a-94c1-4c57-a4d5-b1169813eb77" containerName="istio-proxy" containerID="cri-o://3876a2a0d88e0222c809ec4aa79b49a37b10bb53bdb06c92e3ebf5bfac02d626" gracePeriod=30 Jul 31 19:28:21.248736 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:21.248706 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:28:21.526531 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:21.526497 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:28:21.928663 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:21.928560 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:28:22.230286 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.230188 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw"] Jul 31 19:28:22.230764 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.230745 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="976f6576-26f5-466a-ac26-a861489143f5" containerName="istio-proxy" Jul 31 19:28:22.230814 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.230769 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="976f6576-26f5-466a-ac26-a861489143f5" containerName="istio-proxy" Jul 31 19:28:22.230814 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.230793 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="976f6576-26f5-466a-ac26-a861489143f5" containerName="istio-proxy" Jul 31 19:28:22.230814 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.230801 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="976f6576-26f5-466a-ac26-a861489143f5" containerName="istio-proxy" Jul 31 19:28:22.230907 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.230835 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28969727-0efa-4615-a4fb-29ffb44a6b78" containerName="revoke-keys" Jul 31 19:28:22.230907 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.230844 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="28969727-0efa-4615-a4fb-29ffb44a6b78" containerName="revoke-keys" Jul 31 19:28:22.230973 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.230928 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="976f6576-26f5-466a-ac26-a861489143f5" containerName="istio-proxy" Jul 31 19:28:22.230973 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.230939 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="28969727-0efa-4615-a4fb-29ffb44a6b78" containerName="revoke-keys" Jul 31 19:28:22.230973 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.230950 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="976f6576-26f5-466a-ac26-a861489143f5" containerName="istio-proxy" Jul 31 19:28:22.235963 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.235943 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:22.240093 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.240077 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-fd0284-openshift-default-dockercfg-fflwx\"" Jul 31 19:28:22.271511 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.271483 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw"] Jul 31 19:28:22.309501 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.309464 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:28:22.310629 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.310608 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f0b2bc3-d509-461c-90a5-0d68eb70e012-workload-certs\") pod \"e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:22.310701 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.310643 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f0b2bc3-d509-461c-90a5-0d68eb70e012-istiod-ca-cert\") pod \"e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:22.310701 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.310669 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f0b2bc3-d509-461c-90a5-0d68eb70e012-istio-token\") pod \"e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:22.310800 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.310763 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzhv7\" (UniqueName: \"kubernetes.io/projected/3f0b2bc3-d509-461c-90a5-0d68eb70e012-kube-api-access-kzhv7\") pod \"e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:22.310800 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.310790 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f0b2bc3-d509-461c-90a5-0d68eb70e012-istio-data\") pod \"e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:22.310882 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.310812 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f0b2bc3-d509-461c-90a5-0d68eb70e012-istio-podinfo\") pod \"e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:22.310882 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.310877 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f0b2bc3-d509-461c-90a5-0d68eb70e012-istio-envoy\") pod \"e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:22.310954 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.310932 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0b2bc3-d509-461c-90a5-0d68eb70e012-workload-socket\") pod \"e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:22.310995 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.310957 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0b2bc3-d509-461c-90a5-0d68eb70e012-credential-socket\") pod \"e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:22.412372 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.412335 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f0b2bc3-d509-461c-90a5-0d68eb70e012-workload-certs\") pod \"e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:22.412579 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.412385 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f0b2bc3-d509-461c-90a5-0d68eb70e012-istiod-ca-cert\") pod \"e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:22.412579 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.412427 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f0b2bc3-d509-461c-90a5-0d68eb70e012-istio-token\") pod \"e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:22.412717 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.412572 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzhv7\" (UniqueName: \"kubernetes.io/projected/3f0b2bc3-d509-461c-90a5-0d68eb70e012-kube-api-access-kzhv7\") pod \"e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:22.412717 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.412632 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f0b2bc3-d509-461c-90a5-0d68eb70e012-istio-data\") pod \"e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:22.412717 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.412668 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f0b2bc3-d509-461c-90a5-0d68eb70e012-istio-podinfo\") pod \"e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:22.412872 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.412721 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f0b2bc3-d509-461c-90a5-0d68eb70e012-istio-envoy\") pod \"e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:22.412872 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.412753 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f0b2bc3-d509-461c-90a5-0d68eb70e012-workload-certs\") pod \"e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:22.412872 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.412813 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0b2bc3-d509-461c-90a5-0d68eb70e012-workload-socket\") pod \"e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:22.412872 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.412850 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0b2bc3-d509-461c-90a5-0d68eb70e012-credential-socket\") pod \"e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:22.413090 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.412918 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f0b2bc3-d509-461c-90a5-0d68eb70e012-istio-data\") pod \"e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:22.413148 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.413126 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0b2bc3-d509-461c-90a5-0d68eb70e012-credential-socket\") pod \"e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:22.413210 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.413190 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0b2bc3-d509-461c-90a5-0d68eb70e012-workload-socket\") pod \"e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:22.413287 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.413192 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f0b2bc3-d509-461c-90a5-0d68eb70e012-istiod-ca-cert\") pod \"e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:22.415035 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.415013 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f0b2bc3-d509-461c-90a5-0d68eb70e012-istio-envoy\") pod \"e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:22.415145 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.415127 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f0b2bc3-d509-461c-90a5-0d68eb70e012-istio-podinfo\") pod \"e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:22.421560 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.421536 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzhv7\" (UniqueName: \"kubernetes.io/projected/3f0b2bc3-d509-461c-90a5-0d68eb70e012-kube-api-access-kzhv7\") pod \"e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:22.421671 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.421615 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f0b2bc3-d509-461c-90a5-0d68eb70e012-istio-token\") pod \"e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:22.489117 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.488998 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:28:22.545915 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.545889 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:22.696813 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.696783 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw"] Jul 31 19:28:22.699877 ip-10-0-139-39 kubenswrapper[2565]: W0731 19:28:22.699837 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f0b2bc3_d509_461c_90a5_0d68eb70e012.slice/crio-ed23778600ed1e4e8ca20dfe82ffa968fc9dd78321d9d1b6c3e99f60a426507b WatchSource:0}: Error finding container ed23778600ed1e4e8ca20dfe82ffa968fc9dd78321d9d1b6c3e99f60a426507b: Status 404 returned error can't find the container with id ed23778600ed1e4e8ca20dfe82ffa968fc9dd78321d9d1b6c3e99f60a426507b Jul 31 19:28:22.702355 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.702322 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:28:22.702455 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.702405 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:28:22.702518 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:22.702460 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:28:23.272863 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:23.272825 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" event={"ID":"3f0b2bc3-d509-461c-90a5-0d68eb70e012","Type":"ContainerStarted","Data":"c81137c96b254afdc7a6524989e0eabb780924be24e268e7e3d8270a5fd64571"} Jul 31 19:28:23.273306 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:23.272876 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" event={"ID":"3f0b2bc3-d509-461c-90a5-0d68eb70e012","Type":"ContainerStarted","Data":"ed23778600ed1e4e8ca20dfe82ffa968fc9dd78321d9d1b6c3e99f60a426507b"} Jul 31 19:28:23.293505 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:23.293454 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" podStartSLOduration=1.293436212 podStartE2EDuration="1.293436212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:28:23.292825362 +0000 UTC m=+2679.930815163" watchObservedRunningTime="2026-07-31 19:28:23.293436212 +0000 UTC m=+2679.931426011" Jul 31 19:28:23.547179 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:23.547096 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:23.552062 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:23.552035 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:24.277794 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:24.277767 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:24.278596 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:24.278575 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:25.811380 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:25.811346 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:28:25.911480 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:25.911445 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:28:26.470798 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:26.470769 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:28:26.553708 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:26.553672 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7085d14a-94c1-4c57-a4d5-b1169813eb77-credential-socket\") pod \"7085d14a-94c1-4c57-a4d5-b1169813eb77\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " Jul 31 19:28:26.553708 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:26.553717 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7085d14a-94c1-4c57-a4d5-b1169813eb77-workload-socket\") pod \"7085d14a-94c1-4c57-a4d5-b1169813eb77\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " Jul 31 19:28:26.553978 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:26.553745 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7085d14a-94c1-4c57-a4d5-b1169813eb77-istiod-ca-cert\") pod \"7085d14a-94c1-4c57-a4d5-b1169813eb77\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " Jul 31 19:28:26.553978 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:26.553768 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7085d14a-94c1-4c57-a4d5-b1169813eb77-istio-podinfo\") pod \"7085d14a-94c1-4c57-a4d5-b1169813eb77\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " Jul 31 19:28:26.553978 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:26.553793 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89lcw\" (UniqueName: \"kubernetes.io/projected/7085d14a-94c1-4c57-a4d5-b1169813eb77-kube-api-access-89lcw\") pod \"7085d14a-94c1-4c57-a4d5-b1169813eb77\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " Jul 31 19:28:26.553978 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:26.553818 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7085d14a-94c1-4c57-a4d5-b1169813eb77-istio-token\") pod \"7085d14a-94c1-4c57-a4d5-b1169813eb77\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " Jul 31 19:28:26.553978 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:26.553842 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7085d14a-94c1-4c57-a4d5-b1169813eb77-workload-certs\") pod \"7085d14a-94c1-4c57-a4d5-b1169813eb77\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " Jul 31 19:28:26.553978 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:26.553919 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7085d14a-94c1-4c57-a4d5-b1169813eb77-istio-data\") pod \"7085d14a-94c1-4c57-a4d5-b1169813eb77\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " Jul 31 19:28:26.553978 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:26.553957 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7085d14a-94c1-4c57-a4d5-b1169813eb77-istio-envoy\") pod \"7085d14a-94c1-4c57-a4d5-b1169813eb77\" (UID: \"7085d14a-94c1-4c57-a4d5-b1169813eb77\") " Jul 31 19:28:26.554380 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:26.553991 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7085d14a-94c1-4c57-a4d5-b1169813eb77-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7085d14a-94c1-4c57-a4d5-b1169813eb77" (UID: "7085d14a-94c1-4c57-a4d5-b1169813eb77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:28:26.554380 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:26.554146 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7085d14a-94c1-4c57-a4d5-b1169813eb77-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7085d14a-94c1-4c57-a4d5-b1169813eb77" (UID: "7085d14a-94c1-4c57-a4d5-b1169813eb77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 19:28:26.554380 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:26.554267 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7085d14a-94c1-4c57-a4d5-b1169813eb77-istio-data" (OuterVolumeSpecName: "istio-data") pod "7085d14a-94c1-4c57-a4d5-b1169813eb77" (UID: "7085d14a-94c1-4c57-a4d5-b1169813eb77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:28:26.554380 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:26.554335 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7085d14a-94c1-4c57-a4d5-b1169813eb77-workload-socket\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:28:26.554380 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:26.554352 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7085d14a-94c1-4c57-a4d5-b1169813eb77-istiod-ca-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:28:26.554611 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:26.554408 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7085d14a-94c1-4c57-a4d5-b1169813eb77-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7085d14a-94c1-4c57-a4d5-b1169813eb77" (UID: "7085d14a-94c1-4c57-a4d5-b1169813eb77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:28:26.554611 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:26.554486 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7085d14a-94c1-4c57-a4d5-b1169813eb77-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7085d14a-94c1-4c57-a4d5-b1169813eb77" (UID: "7085d14a-94c1-4c57-a4d5-b1169813eb77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:28:26.556124 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:26.556095 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7085d14a-94c1-4c57-a4d5-b1169813eb77-istio-token" (OuterVolumeSpecName: "istio-token") pod "7085d14a-94c1-4c57-a4d5-b1169813eb77" (UID: "7085d14a-94c1-4c57-a4d5-b1169813eb77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:28:26.556223 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:26.556199 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7085d14a-94c1-4c57-a4d5-b1169813eb77-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7085d14a-94c1-4c57-a4d5-b1169813eb77" (UID: "7085d14a-94c1-4c57-a4d5-b1169813eb77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 19:28:26.556542 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:26.556514 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7085d14a-94c1-4c57-a4d5-b1169813eb77-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7085d14a-94c1-4c57-a4d5-b1169813eb77" (UID: "7085d14a-94c1-4c57-a4d5-b1169813eb77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:28:26.556542 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:26.556518 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7085d14a-94c1-4c57-a4d5-b1169813eb77-kube-api-access-89lcw" (OuterVolumeSpecName: "kube-api-access-89lcw") pod "7085d14a-94c1-4c57-a4d5-b1169813eb77" (UID: "7085d14a-94c1-4c57-a4d5-b1169813eb77"). InnerVolumeSpecName "kube-api-access-89lcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:28:26.655763 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:26.655723 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7085d14a-94c1-4c57-a4d5-b1169813eb77-istio-data\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:28:26.655763 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:26.655754 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7085d14a-94c1-4c57-a4d5-b1169813eb77-istio-envoy\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:28:26.655763 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:26.655764 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7085d14a-94c1-4c57-a4d5-b1169813eb77-credential-socket\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:28:26.655763 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:26.655773 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7085d14a-94c1-4c57-a4d5-b1169813eb77-istio-podinfo\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:28:26.656028 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:26.655782 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89lcw\" (UniqueName: \"kubernetes.io/projected/7085d14a-94c1-4c57-a4d5-b1169813eb77-kube-api-access-89lcw\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:28:26.656028 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:26.655791 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7085d14a-94c1-4c57-a4d5-b1169813eb77-istio-token\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:28:26.656028 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:26.655799 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7085d14a-94c1-4c57-a4d5-b1169813eb77-workload-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:28:26.930609 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:26.930508 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:28:27.296582 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:27.296545 2565 generic.go:356] "Generic (PLEG): container finished" podID="7085d14a-94c1-4c57-a4d5-b1169813eb77" containerID="3876a2a0d88e0222c809ec4aa79b49a37b10bb53bdb06c92e3ebf5bfac02d626" exitCode=0 Jul 31 19:28:27.296781 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:27.296590 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" event={"ID":"7085d14a-94c1-4c57-a4d5-b1169813eb77","Type":"ContainerDied","Data":"3876a2a0d88e0222c809ec4aa79b49a37b10bb53bdb06c92e3ebf5bfac02d626"} Jul 31 19:28:27.296781 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:27.296597 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" Jul 31 19:28:27.296781 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:27.296638 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9" event={"ID":"7085d14a-94c1-4c57-a4d5-b1169813eb77","Type":"ContainerDied","Data":"1cb76ef03b7c2cf66d993dff7b071fbbe50f9463454306e503ce6cc2c98706c2"} Jul 31 19:28:27.296781 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:27.296661 2565 scope.go:117] "RemoveContainer" containerID="3876a2a0d88e0222c809ec4aa79b49a37b10bb53bdb06c92e3ebf5bfac02d626" Jul 31 19:28:27.306393 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:27.306378 2565 scope.go:117] "RemoveContainer" containerID="ddc519c7f80f576cd45d5449fc4cb9582a34e325a58d88951abbafe30ccfc985" Jul 31 19:28:27.314406 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:27.314386 2565 scope.go:117] "RemoveContainer" containerID="3876a2a0d88e0222c809ec4aa79b49a37b10bb53bdb06c92e3ebf5bfac02d626" Jul 31 19:28:27.314642 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:28:27.314623 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3876a2a0d88e0222c809ec4aa79b49a37b10bb53bdb06c92e3ebf5bfac02d626\": container with ID starting with 3876a2a0d88e0222c809ec4aa79b49a37b10bb53bdb06c92e3ebf5bfac02d626 not found: ID does not exist" containerID="3876a2a0d88e0222c809ec4aa79b49a37b10bb53bdb06c92e3ebf5bfac02d626" Jul 31 19:28:27.314693 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:27.314651 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3876a2a0d88e0222c809ec4aa79b49a37b10bb53bdb06c92e3ebf5bfac02d626"} err="failed to get container status \"3876a2a0d88e0222c809ec4aa79b49a37b10bb53bdb06c92e3ebf5bfac02d626\": rpc error: code = NotFound desc = could not find container \"3876a2a0d88e0222c809ec4aa79b49a37b10bb53bdb06c92e3ebf5bfac02d626\": container with ID starting with 3876a2a0d88e0222c809ec4aa79b49a37b10bb53bdb06c92e3ebf5bfac02d626 not found: ID does not exist" Jul 31 19:28:27.314693 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:27.314669 2565 scope.go:117] "RemoveContainer" containerID="ddc519c7f80f576cd45d5449fc4cb9582a34e325a58d88951abbafe30ccfc985" Jul 31 19:28:27.314873 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:28:27.314857 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddc519c7f80f576cd45d5449fc4cb9582a34e325a58d88951abbafe30ccfc985\": container with ID starting with ddc519c7f80f576cd45d5449fc4cb9582a34e325a58d88951abbafe30ccfc985 not found: ID does not exist" containerID="ddc519c7f80f576cd45d5449fc4cb9582a34e325a58d88951abbafe30ccfc985" Jul 31 19:28:27.314910 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:27.314878 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddc519c7f80f576cd45d5449fc4cb9582a34e325a58d88951abbafe30ccfc985"} err="failed to get container status \"ddc519c7f80f576cd45d5449fc4cb9582a34e325a58d88951abbafe30ccfc985\": rpc error: code = NotFound desc = could not find container \"ddc519c7f80f576cd45d5449fc4cb9582a34e325a58d88951abbafe30ccfc985\": container with ID starting with ddc519c7f80f576cd45d5449fc4cb9582a34e325a58d88951abbafe30ccfc985 not found: ID does not exist" Jul 31 19:28:27.319608 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:27.319587 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9"] Jul 31 19:28:27.323512 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:27.323493 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f3dafe-openshift-default-6649dfd876-h9zf9"] Jul 31 19:28:27.452106 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:27.452040 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:28:27.551964 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:27.551884 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:28:27.887164 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:27.887090 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7085d14a-94c1-4c57-a4d5-b1169813eb77" path="/var/lib/kubelet/pods/7085d14a-94c1-4c57-a4d5-b1169813eb77/volumes" Jul 31 19:28:29.450805 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:29.450769 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:28:39.279634 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:39.279595 2565 patch_prober.go:28] interesting pod/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 31 19:28:39.280054 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:39.279666 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" podUID="3f0b2bc3-d509-461c-90a5-0d68eb70e012" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 31 19:28:39.374439 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:39.374407 2565 generic.go:356] "Generic (PLEG): container finished" podID="3f0b2bc3-d509-461c-90a5-0d68eb70e012" containerID="c81137c96b254afdc7a6524989e0eabb780924be24e268e7e3d8270a5fd64571" exitCode=0 Jul 31 19:28:39.374603 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:39.374455 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" event={"ID":"3f0b2bc3-d509-461c-90a5-0d68eb70e012","Type":"ContainerDied","Data":"c81137c96b254afdc7a6524989e0eabb780924be24e268e7e3d8270a5fd64571"} Jul 31 19:28:39.374852 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:39.374839 2565 scope.go:117] "RemoveContainer" containerID="c81137c96b254afdc7a6524989e0eabb780924be24e268e7e3d8270a5fd64571" Jul 31 19:28:39.375834 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:39.375805 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:28:39.375894 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:39.375875 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:28:39.375933 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:39.375903 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:28:40.381746 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:40.381709 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" event={"ID":"3f0b2bc3-d509-461c-90a5-0d68eb70e012","Type":"ContainerStarted","Data":"7cc703fc8c932fd59f0647736d45a8d6cb558093975e32ae854e89beafe77b6e"} Jul 31 19:28:40.546947 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:40.546912 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:40.547321 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:40.547298 2565 patch_prober.go:28] interesting pod/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 31 19:28:40.547510 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:40.547485 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" podUID="3f0b2bc3-d509-461c-90a5-0d68eb70e012" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 31 19:28:41.546657 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:41.546610 2565 patch_prober.go:28] interesting pod/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 31 19:28:41.547164 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:41.546680 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" podUID="3f0b2bc3-d509-461c-90a5-0d68eb70e012" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 31 19:28:42.547096 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:42.546642 2565 patch_prober.go:28] interesting pod/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 31 19:28:42.547096 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:42.546707 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" podUID="3f0b2bc3-d509-461c-90a5-0d68eb70e012" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 31 19:28:43.546549 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:43.546512 2565 patch_prober.go:28] interesting pod/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 31 19:28:43.546850 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:43.546819 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" podUID="3f0b2bc3-d509-461c-90a5-0d68eb70e012" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 31 19:28:44.547171 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:44.547129 2565 patch_prober.go:28] interesting pod/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 31 19:28:44.547863 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:44.547829 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" podUID="3f0b2bc3-d509-461c-90a5-0d68eb70e012" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 31 19:28:45.550957 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:45.550922 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:45.551409 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:45.551269 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:45.551956 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:45.551936 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:28:55.911879 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:55.911838 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:28:56.514284 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.514231 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2"] Jul 31 19:28:56.514639 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.514626 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7085d14a-94c1-4c57-a4d5-b1169813eb77" containerName="istio-proxy" Jul 31 19:28:56.514698 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.514640 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7085d14a-94c1-4c57-a4d5-b1169813eb77" containerName="istio-proxy" Jul 31 19:28:56.514698 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.514655 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7085d14a-94c1-4c57-a4d5-b1169813eb77" containerName="istio-proxy" Jul 31 19:28:56.514698 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.514660 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7085d14a-94c1-4c57-a4d5-b1169813eb77" containerName="istio-proxy" Jul 31 19:28:56.514803 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.514713 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7085d14a-94c1-4c57-a4d5-b1169813eb77" containerName="istio-proxy" Jul 31 19:28:56.514803 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.514723 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7085d14a-94c1-4c57-a4d5-b1169813eb77" containerName="istio-proxy" Jul 31 19:28:56.516894 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.516873 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:56.518862 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.518840 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-09a154-openshift-default-dockercfg-ngdlp\"" Jul 31 19:28:56.528561 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.528535 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2"] Jul 31 19:28:56.559411 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.559378 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edcd39a9-332e-42bf-b284-a49014964bb6-istio-data\") pod \"e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:56.559579 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.559416 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edcd39a9-332e-42bf-b284-a49014964bb6-istio-token\") pod \"e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:56.559579 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.559437 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edcd39a9-332e-42bf-b284-a49014964bb6-istio-envoy\") pod \"e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:56.559579 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.559476 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edcd39a9-332e-42bf-b284-a49014964bb6-istiod-ca-cert\") pod \"e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:56.559579 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.559519 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcd39a9-332e-42bf-b284-a49014964bb6-credential-socket\") pod \"e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:56.559812 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.559583 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcd39a9-332e-42bf-b284-a49014964bb6-workload-socket\") pod \"e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:56.559812 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.559643 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22z42\" (UniqueName: \"kubernetes.io/projected/edcd39a9-332e-42bf-b284-a49014964bb6-kube-api-access-22z42\") pod \"e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:56.559812 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.559666 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edcd39a9-332e-42bf-b284-a49014964bb6-istio-podinfo\") pod \"e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:56.559812 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.559691 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edcd39a9-332e-42bf-b284-a49014964bb6-workload-certs\") pod \"e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:56.660505 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.660469 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edcd39a9-332e-42bf-b284-a49014964bb6-istio-envoy\") pod \"e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:56.660505 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.660505 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edcd39a9-332e-42bf-b284-a49014964bb6-istiod-ca-cert\") pod \"e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:56.660766 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.660529 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcd39a9-332e-42bf-b284-a49014964bb6-credential-socket\") pod \"e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:56.660766 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.660568 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcd39a9-332e-42bf-b284-a49014964bb6-workload-socket\") pod \"e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:56.660766 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.660631 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22z42\" (UniqueName: \"kubernetes.io/projected/edcd39a9-332e-42bf-b284-a49014964bb6-kube-api-access-22z42\") pod \"e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:56.660766 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.660659 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edcd39a9-332e-42bf-b284-a49014964bb6-istio-podinfo\") pod \"e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:56.660766 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.660689 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edcd39a9-332e-42bf-b284-a49014964bb6-workload-certs\") pod \"e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:56.660766 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.660748 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edcd39a9-332e-42bf-b284-a49014964bb6-istio-data\") pod \"e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:56.661082 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.660788 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edcd39a9-332e-42bf-b284-a49014964bb6-istio-token\") pod \"e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:56.661082 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.661033 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcd39a9-332e-42bf-b284-a49014964bb6-workload-socket\") pod \"e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:56.661184 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.661086 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcd39a9-332e-42bf-b284-a49014964bb6-credential-socket\") pod \"e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:56.661270 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.661198 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edcd39a9-332e-42bf-b284-a49014964bb6-workload-certs\") pod \"e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:56.661270 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.661217 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edcd39a9-332e-42bf-b284-a49014964bb6-istio-data\") pod \"e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:56.661489 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.661468 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edcd39a9-332e-42bf-b284-a49014964bb6-istiod-ca-cert\") pod \"e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:56.662886 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.662865 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edcd39a9-332e-42bf-b284-a49014964bb6-istio-envoy\") pod \"e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:56.663172 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.663150 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edcd39a9-332e-42bf-b284-a49014964bb6-istio-podinfo\") pod \"e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:56.668948 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.668926 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edcd39a9-332e-42bf-b284-a49014964bb6-istio-token\") pod \"e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:56.669096 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.669079 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22z42\" (UniqueName: \"kubernetes.io/projected/edcd39a9-332e-42bf-b284-a49014964bb6-kube-api-access-22z42\") pod \"e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:56.828169 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.828065 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:56.969465 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.969439 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2"] Jul 31 19:28:56.972157 ip-10-0-139-39 kubenswrapper[2565]: W0731 19:28:56.972132 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedcd39a9_332e_42bf_b284_a49014964bb6.slice/crio-ac8a9144135672a258470c7bb6f69d874a56033cbc7e1cbf56f1a9eedc3accfc WatchSource:0}: Error finding container ac8a9144135672a258470c7bb6f69d874a56033cbc7e1cbf56f1a9eedc3accfc: Status 404 returned error can't find the container with id ac8a9144135672a258470c7bb6f69d874a56033cbc7e1cbf56f1a9eedc3accfc Jul 31 19:28:56.974749 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.974720 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:28:56.974828 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.974782 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:28:56.974828 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:56.974808 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:28:57.473913 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:57.473878 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" event={"ID":"edcd39a9-332e-42bf-b284-a49014964bb6","Type":"ContainerStarted","Data":"6d517e32997a47a436d5be6a8183a4b609c330d2eed44d02d95c8b7b229f15f2"} Jul 31 19:28:57.474101 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:57.473936 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" event={"ID":"edcd39a9-332e-42bf-b284-a49014964bb6","Type":"ContainerStarted","Data":"ac8a9144135672a258470c7bb6f69d874a56033cbc7e1cbf56f1a9eedc3accfc"} Jul 31 19:28:57.482690 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:57.482664 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:28:57.497399 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:57.496714 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" podStartSLOduration=1.4966957189999999 podStartE2EDuration="1.496695719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:28:57.493652213 +0000 UTC m=+2714.131642014" watchObservedRunningTime="2026-07-31 19:28:57.496695719 +0000 UTC m=+2714.134685519" Jul 31 19:28:57.828904 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:57.828864 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:58.834423 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:58.834389 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:59.431330 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:59.431295 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:28:59.483298 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:59.483264 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:59.484065 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:59.484047 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:28:59.513717 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:28:59.513680 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:29:00.558555 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:00.558514 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:29:01.064935 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:01.064898 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:29:01.190267 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:01.190211 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:29:01.364805 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:01.364722 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:29:03.360043 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:03.360009 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:29:13.568273 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:13.568225 2565 generic.go:356] "Generic (PLEG): container finished" podID="edcd39a9-332e-42bf-b284-a49014964bb6" containerID="6d517e32997a47a436d5be6a8183a4b609c330d2eed44d02d95c8b7b229f15f2" exitCode=0 Jul 31 19:29:13.568711 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:13.568297 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" event={"ID":"edcd39a9-332e-42bf-b284-a49014964bb6","Type":"ContainerDied","Data":"6d517e32997a47a436d5be6a8183a4b609c330d2eed44d02d95c8b7b229f15f2"} Jul 31 19:29:13.568768 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:13.568742 2565 scope.go:117] "RemoveContainer" containerID="6d517e32997a47a436d5be6a8183a4b609c330d2eed44d02d95c8b7b229f15f2" Jul 31 19:29:13.569776 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:13.569751 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:29:13.569869 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:13.569855 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:29:13.569911 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:13.569891 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:29:14.484091 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:14.484050 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:29:14.575066 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:14.575030 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" event={"ID":"edcd39a9-332e-42bf-b284-a49014964bb6","Type":"ContainerStarted","Data":"507ebc11edecbb79291f73ab453efee9393293731bd9f3b90b5382acd16c0f99"} Jul 31 19:29:14.828866 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:14.828829 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:29:14.830090 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:14.830060 2565 patch_prober.go:28] interesting pod/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 31 19:29:14.830219 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:14.830137 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" podUID="edcd39a9-332e-42bf-b284-a49014964bb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 31 19:29:15.838263 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:15.829399 2565 patch_prober.go:28] interesting pod/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 31 19:29:15.838263 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:15.829464 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" podUID="edcd39a9-332e-42bf-b284-a49014964bb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 31 19:29:16.828607 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:16.828566 2565 patch_prober.go:28] interesting pod/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 31 19:29:16.828830 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:16.828648 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" podUID="edcd39a9-332e-42bf-b284-a49014964bb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 31 19:29:17.828836 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:17.828798 2565 patch_prober.go:28] interesting pod/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 31 19:29:17.829443 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:17.829412 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" podUID="edcd39a9-332e-42bf-b284-a49014964bb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 31 19:29:18.829441 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:18.829400 2565 patch_prober.go:28] interesting pod/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 31 19:29:18.830032 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:18.829996 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" podUID="edcd39a9-332e-42bf-b284-a49014964bb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 31 19:29:19.833946 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:19.833912 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:29:19.834431 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:19.834293 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:29:19.835069 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:19.835052 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:29:24.404156 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:24.404117 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:29:39.388769 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:39.388726 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8"] Jul 31 19:29:39.392049 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:39.392027 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8" Jul 31 19:29:39.394650 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:39.394618 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-fd0284\"/\"kube-root-ca.crt\"" Jul 31 19:29:39.394771 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:39.394692 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-fd0284\"/\"auth-test-model-fd0284-kserve-self-signed-certs\"" Jul 31 19:29:39.394771 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:39.394694 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-fd0284\"/\"default-dockercfg-s5bzh\"" Jul 31 19:29:39.394771 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:39.394721 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-fd0284\"/\"openshift-service-ca.crt\"" Jul 31 19:29:39.402253 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:39.402209 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8"] Jul 31 19:29:39.418332 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:39.418304 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:29:39.459090 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:39.459057 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px4z4\" (UniqueName: \"kubernetes.io/projected/357bc490-b170-466b-abf9-0ca8508cc257-kube-api-access-px4z4\") pod \"auth-test-model-fd0284-kserve-7d64949c47-2gbv8\" (UID: \"357bc490-b170-466b-abf9-0ca8508cc257\") " pod="ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8" Jul 31 19:29:39.459090 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:39.459095 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/357bc490-b170-466b-abf9-0ca8508cc257-home\") pod \"auth-test-model-fd0284-kserve-7d64949c47-2gbv8\" (UID: \"357bc490-b170-466b-abf9-0ca8508cc257\") " pod="ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8" Jul 31 19:29:39.459334 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:39.459114 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/357bc490-b170-466b-abf9-0ca8508cc257-dshm\") pod \"auth-test-model-fd0284-kserve-7d64949c47-2gbv8\" (UID: \"357bc490-b170-466b-abf9-0ca8508cc257\") " pod="ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8" Jul 31 19:29:39.459334 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:39.459130 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/357bc490-b170-466b-abf9-0ca8508cc257-tls-certs\") pod \"auth-test-model-fd0284-kserve-7d64949c47-2gbv8\" (UID: \"357bc490-b170-466b-abf9-0ca8508cc257\") " pod="ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8" Jul 31 19:29:39.459334 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:39.459195 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/357bc490-b170-466b-abf9-0ca8508cc257-tmp-dir\") pod \"auth-test-model-fd0284-kserve-7d64949c47-2gbv8\" (UID: \"357bc490-b170-466b-abf9-0ca8508cc257\") " pod="ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8" Jul 31 19:29:39.459334 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:39.459325 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/357bc490-b170-466b-abf9-0ca8508cc257-model-cache\") pod \"auth-test-model-fd0284-kserve-7d64949c47-2gbv8\" (UID: \"357bc490-b170-466b-abf9-0ca8508cc257\") " pod="ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8" Jul 31 19:29:39.560227 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:39.560189 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/357bc490-b170-466b-abf9-0ca8508cc257-model-cache\") pod \"auth-test-model-fd0284-kserve-7d64949c47-2gbv8\" (UID: \"357bc490-b170-466b-abf9-0ca8508cc257\") " pod="ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8" Jul 31 19:29:39.560227 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:39.560247 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px4z4\" (UniqueName: \"kubernetes.io/projected/357bc490-b170-466b-abf9-0ca8508cc257-kube-api-access-px4z4\") pod \"auth-test-model-fd0284-kserve-7d64949c47-2gbv8\" (UID: \"357bc490-b170-466b-abf9-0ca8508cc257\") " pod="ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8" Jul 31 19:29:39.560498 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:39.560270 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/357bc490-b170-466b-abf9-0ca8508cc257-home\") pod \"auth-test-model-fd0284-kserve-7d64949c47-2gbv8\" (UID: \"357bc490-b170-466b-abf9-0ca8508cc257\") " pod="ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8" Jul 31 19:29:39.560498 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:39.560285 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/357bc490-b170-466b-abf9-0ca8508cc257-dshm\") pod \"auth-test-model-fd0284-kserve-7d64949c47-2gbv8\" (UID: \"357bc490-b170-466b-abf9-0ca8508cc257\") " pod="ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8" Jul 31 19:29:39.560498 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:39.560302 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/357bc490-b170-466b-abf9-0ca8508cc257-tls-certs\") pod \"auth-test-model-fd0284-kserve-7d64949c47-2gbv8\" (UID: \"357bc490-b170-466b-abf9-0ca8508cc257\") " pod="ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8" Jul 31 19:29:39.560498 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:39.560426 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/357bc490-b170-466b-abf9-0ca8508cc257-tmp-dir\") pod \"auth-test-model-fd0284-kserve-7d64949c47-2gbv8\" (UID: \"357bc490-b170-466b-abf9-0ca8508cc257\") " pod="ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8" Jul 31 19:29:39.560740 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:39.560690 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/357bc490-b170-466b-abf9-0ca8508cc257-model-cache\") pod \"auth-test-model-fd0284-kserve-7d64949c47-2gbv8\" (UID: \"357bc490-b170-466b-abf9-0ca8508cc257\") " pod="ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8" Jul 31 19:29:39.560740 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:39.560728 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/357bc490-b170-466b-abf9-0ca8508cc257-home\") pod \"auth-test-model-fd0284-kserve-7d64949c47-2gbv8\" (UID: \"357bc490-b170-466b-abf9-0ca8508cc257\") " pod="ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8" Jul 31 19:29:39.560822 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:39.560804 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/357bc490-b170-466b-abf9-0ca8508cc257-tmp-dir\") pod \"auth-test-model-fd0284-kserve-7d64949c47-2gbv8\" (UID: \"357bc490-b170-466b-abf9-0ca8508cc257\") " pod="ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8" Jul 31 19:29:39.562548 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:39.562520 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/357bc490-b170-466b-abf9-0ca8508cc257-dshm\") pod \"auth-test-model-fd0284-kserve-7d64949c47-2gbv8\" (UID: \"357bc490-b170-466b-abf9-0ca8508cc257\") " pod="ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8" Jul 31 19:29:39.562818 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:39.562802 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/357bc490-b170-466b-abf9-0ca8508cc257-tls-certs\") pod \"auth-test-model-fd0284-kserve-7d64949c47-2gbv8\" (UID: \"357bc490-b170-466b-abf9-0ca8508cc257\") " pod="ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8" Jul 31 19:29:39.568908 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:39.568885 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px4z4\" (UniqueName: \"kubernetes.io/projected/357bc490-b170-466b-abf9-0ca8508cc257-kube-api-access-px4z4\") pod \"auth-test-model-fd0284-kserve-7d64949c47-2gbv8\" (UID: \"357bc490-b170-466b-abf9-0ca8508cc257\") " pod="ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8" Jul 31 19:29:39.702653 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:39.702566 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8" Jul 31 19:29:40.044928 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:40.044867 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8"] Jul 31 19:29:40.047188 ip-10-0-139-39 kubenswrapper[2565]: W0731 19:29:40.047156 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod357bc490_b170_466b_abf9_0ca8508cc257.slice/crio-93ac1bb2f700f26e1507b959f6191508e19fd10cb60950f912d893d29901c13f WatchSource:0}: Error finding container 93ac1bb2f700f26e1507b959f6191508e19fd10cb60950f912d893d29901c13f: Status 404 returned error can't find the container with id 93ac1bb2f700f26e1507b959f6191508e19fd10cb60950f912d893d29901c13f Jul 31 19:29:40.707784 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:40.707747 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8" event={"ID":"357bc490-b170-466b-abf9-0ca8508cc257","Type":"ContainerStarted","Data":"da20e2eefff32c49fe934e534cfd67470df4a51ca1cf6af7f5aae529372e855a"} Jul 31 19:29:40.707784 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:40.707789 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8" event={"ID":"357bc490-b170-466b-abf9-0ca8508cc257","Type":"ContainerStarted","Data":"93ac1bb2f700f26e1507b959f6191508e19fd10cb60950f912d893d29901c13f"} Jul 31 19:29:40.727466 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:40.727403 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8" podStartSLOduration=1.727382944 podStartE2EDuration="1.727382944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:29:40.724577111 +0000 UTC m=+2757.362566912" watchObservedRunningTime="2026-07-31 19:29:40.727382944 +0000 UTC m=+2757.365372745" Jul 31 19:29:41.747982 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:41.747949 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:29:49.703473 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:49.703438 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8" Jul 31 19:29:49.703978 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:49.703489 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8" Jul 31 19:29:49.716113 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:49.716087 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8" Jul 31 19:29:49.762852 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:49.762823 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8" Jul 31 19:29:51.659209 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:29:51.659171 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:30:08.823762 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:30:08.823690 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:30:11.465835 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:30:11.465796 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:30:34.964669 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:30:34.964632 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:30:44.195305 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:30:44.195273 2565 scope.go:117] "RemoveContainer" containerID="0a73315d8fe3a176f7bc84e0997c40ca68900b2b618565ea97f8adc73fed7f78" Jul 31 19:30:50.348627 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:30:50.348585 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:31:00.760708 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:31:00.760663 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:31:16.656703 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:31:16.656665 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:31:30.055889 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:31:30.055853 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:31:38.063398 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:31:38.063316 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:31:46.853409 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:31:46.853374 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:31:57.350318 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:31:57.350278 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:32:08.756592 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:32:08.756556 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:32:14.874416 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:32:14.874380 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:32:14.968211 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:32:14.968170 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:32:23.957010 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:32:23.956973 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:32:33.860021 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:32:33.859980 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:32:50.359324 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:32:50.359283 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:33:05.854154 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:33:05.854119 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:33:11.201190 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:33:11.201105 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:33:16.456231 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:33:16.456190 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:33:26.403379 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:33:26.403338 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:33:32.057515 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:33:32.057477 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:33:44.228973 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:33:44.228938 2565 scope.go:117] "RemoveContainer" containerID="1f31172d9ec76fdb0778d40358e45b06e664abd1811ed55608e5876f2bd8f453" Jul 31 19:33:47.754546 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:33:47.754511 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:33:51.548963 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:33:51.548924 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8"] Jul 31 19:33:51.569566 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:33:51.569527 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:33:52.498506 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:33:52.498470 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:33:57.958995 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:33:57.958955 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:34:00.803682 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:00.803648 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x"] Jul 31 19:34:00.806534 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:00.806513 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x" Jul 31 19:34:00.808463 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:00.808440 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-fd0284\"/\"sub-test-model-fd0284-kserve-self-signed-certs\"" Jul 31 19:34:00.816938 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:00.816915 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x"] Jul 31 19:34:00.831844 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:00.831820 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:34:00.896486 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:00.896457 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/888aeb55-6bb3-43ff-bce5-c70224451840-tmp-dir\") pod \"sub-test-model-fd0284-kserve-7dd5d64b95-lm24x\" (UID: \"888aeb55-6bb3-43ff-bce5-c70224451840\") " pod="ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x" Jul 31 19:34:00.896664 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:00.896491 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvpqq\" (UniqueName: \"kubernetes.io/projected/888aeb55-6bb3-43ff-bce5-c70224451840-kube-api-access-nvpqq\") pod \"sub-test-model-fd0284-kserve-7dd5d64b95-lm24x\" (UID: \"888aeb55-6bb3-43ff-bce5-c70224451840\") " pod="ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x" Jul 31 19:34:00.896664 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:00.896512 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/888aeb55-6bb3-43ff-bce5-c70224451840-tls-certs\") pod \"sub-test-model-fd0284-kserve-7dd5d64b95-lm24x\" (UID: \"888aeb55-6bb3-43ff-bce5-c70224451840\") " pod="ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x" Jul 31 19:34:00.896664 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:00.896641 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/888aeb55-6bb3-43ff-bce5-c70224451840-dshm\") pod \"sub-test-model-fd0284-kserve-7dd5d64b95-lm24x\" (UID: \"888aeb55-6bb3-43ff-bce5-c70224451840\") " pod="ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x" Jul 31 19:34:00.896781 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:00.896674 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/888aeb55-6bb3-43ff-bce5-c70224451840-home\") pod \"sub-test-model-fd0284-kserve-7dd5d64b95-lm24x\" (UID: \"888aeb55-6bb3-43ff-bce5-c70224451840\") " pod="ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x" Jul 31 19:34:00.896816 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:00.896783 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/888aeb55-6bb3-43ff-bce5-c70224451840-model-cache\") pod \"sub-test-model-fd0284-kserve-7dd5d64b95-lm24x\" (UID: \"888aeb55-6bb3-43ff-bce5-c70224451840\") " pod="ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x" Jul 31 19:34:00.997782 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:00.997738 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/888aeb55-6bb3-43ff-bce5-c70224451840-dshm\") pod \"sub-test-model-fd0284-kserve-7dd5d64b95-lm24x\" (UID: \"888aeb55-6bb3-43ff-bce5-c70224451840\") " pod="ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x" Jul 31 19:34:00.997782 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:00.997781 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/888aeb55-6bb3-43ff-bce5-c70224451840-home\") pod \"sub-test-model-fd0284-kserve-7dd5d64b95-lm24x\" (UID: \"888aeb55-6bb3-43ff-bce5-c70224451840\") " pod="ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x" Jul 31 19:34:00.998000 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:00.997861 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/888aeb55-6bb3-43ff-bce5-c70224451840-model-cache\") pod \"sub-test-model-fd0284-kserve-7dd5d64b95-lm24x\" (UID: \"888aeb55-6bb3-43ff-bce5-c70224451840\") " pod="ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x" Jul 31 19:34:00.998000 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:00.997904 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/888aeb55-6bb3-43ff-bce5-c70224451840-tmp-dir\") pod \"sub-test-model-fd0284-kserve-7dd5d64b95-lm24x\" (UID: \"888aeb55-6bb3-43ff-bce5-c70224451840\") " pod="ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x" Jul 31 19:34:00.998000 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:00.997928 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvpqq\" (UniqueName: \"kubernetes.io/projected/888aeb55-6bb3-43ff-bce5-c70224451840-kube-api-access-nvpqq\") pod \"sub-test-model-fd0284-kserve-7dd5d64b95-lm24x\" (UID: \"888aeb55-6bb3-43ff-bce5-c70224451840\") " pod="ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x" Jul 31 19:34:00.998000 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:00.997950 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/888aeb55-6bb3-43ff-bce5-c70224451840-tls-certs\") pod \"sub-test-model-fd0284-kserve-7dd5d64b95-lm24x\" (UID: \"888aeb55-6bb3-43ff-bce5-c70224451840\") " pod="ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x" Jul 31 19:34:00.998338 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:00.998310 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/888aeb55-6bb3-43ff-bce5-c70224451840-home\") pod \"sub-test-model-fd0284-kserve-7dd5d64b95-lm24x\" (UID: \"888aeb55-6bb3-43ff-bce5-c70224451840\") " pod="ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x" Jul 31 19:34:00.998429 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:00.998361 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/888aeb55-6bb3-43ff-bce5-c70224451840-model-cache\") pod \"sub-test-model-fd0284-kserve-7dd5d64b95-lm24x\" (UID: \"888aeb55-6bb3-43ff-bce5-c70224451840\") " pod="ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x" Jul 31 19:34:00.998429 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:00.998381 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/888aeb55-6bb3-43ff-bce5-c70224451840-tmp-dir\") pod \"sub-test-model-fd0284-kserve-7dd5d64b95-lm24x\" (UID: \"888aeb55-6bb3-43ff-bce5-c70224451840\") " pod="ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x" Jul 31 19:34:01.000056 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:01.000037 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/888aeb55-6bb3-43ff-bce5-c70224451840-dshm\") pod \"sub-test-model-fd0284-kserve-7dd5d64b95-lm24x\" (UID: \"888aeb55-6bb3-43ff-bce5-c70224451840\") " pod="ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x" Jul 31 19:34:01.000346 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:01.000330 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/888aeb55-6bb3-43ff-bce5-c70224451840-tls-certs\") pod \"sub-test-model-fd0284-kserve-7dd5d64b95-lm24x\" (UID: \"888aeb55-6bb3-43ff-bce5-c70224451840\") " pod="ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x" Jul 31 19:34:01.006007 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:01.005983 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvpqq\" (UniqueName: \"kubernetes.io/projected/888aeb55-6bb3-43ff-bce5-c70224451840-kube-api-access-nvpqq\") pod \"sub-test-model-fd0284-kserve-7dd5d64b95-lm24x\" (UID: \"888aeb55-6bb3-43ff-bce5-c70224451840\") " pod="ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x" Jul 31 19:34:01.116809 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:01.116719 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x" Jul 31 19:34:01.255194 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:01.255157 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x"] Jul 31 19:34:01.258160 ip-10-0-139-39 kubenswrapper[2565]: W0731 19:34:01.258134 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod888aeb55_6bb3_43ff_bce5_c70224451840.slice/crio-748b8e18c51cd405304b6b536e34d40fc57ee4ab1bf99cb5d9e2b2eafcdc32da WatchSource:0}: Error finding container 748b8e18c51cd405304b6b536e34d40fc57ee4ab1bf99cb5d9e2b2eafcdc32da: Status 404 returned error can't find the container with id 748b8e18c51cd405304b6b536e34d40fc57ee4ab1bf99cb5d9e2b2eafcdc32da Jul 31 19:34:02.047137 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:02.047103 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x" event={"ID":"888aeb55-6bb3-43ff-bce5-c70224451840","Type":"ContainerStarted","Data":"cdd869c2e95d363866a6c0055052672e6dd6ffe3c9cf00925cd27d0b0424c1c8"} Jul 31 19:34:02.047632 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:02.047145 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x" event={"ID":"888aeb55-6bb3-43ff-bce5-c70224451840","Type":"ContainerStarted","Data":"748b8e18c51cd405304b6b536e34d40fc57ee4ab1bf99cb5d9e2b2eafcdc32da"} Jul 31 19:34:02.065368 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:02.065322 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x" podStartSLOduration=2.065307005 podStartE2EDuration="2.065307005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:34:02.062881812 +0000 UTC m=+3018.700871612" watchObservedRunningTime="2026-07-31 19:34:02.065307005 +0000 UTC m=+3018.703296804" Jul 31 19:34:03.553184 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:03.553150 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:34:06.568531 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:06.568487 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8" podUID="357bc490-b170-466b-abf9-0ca8508cc257" containerName="main" containerID="cri-o://da20e2eefff32c49fe934e534cfd67470df4a51ca1cf6af7f5aae529372e855a" gracePeriod=45 Jul 31 19:34:06.816428 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:06.816401 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8" Jul 31 19:34:06.960143 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:06.960046 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/357bc490-b170-466b-abf9-0ca8508cc257-dshm\") pod \"357bc490-b170-466b-abf9-0ca8508cc257\" (UID: \"357bc490-b170-466b-abf9-0ca8508cc257\") " Jul 31 19:34:06.960143 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:06.960095 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/357bc490-b170-466b-abf9-0ca8508cc257-home\") pod \"357bc490-b170-466b-abf9-0ca8508cc257\" (UID: \"357bc490-b170-466b-abf9-0ca8508cc257\") " Jul 31 19:34:06.960410 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:06.960203 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/357bc490-b170-466b-abf9-0ca8508cc257-model-cache\") pod \"357bc490-b170-466b-abf9-0ca8508cc257\" (UID: \"357bc490-b170-466b-abf9-0ca8508cc257\") " Jul 31 19:34:06.960410 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:06.960257 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-px4z4\" (UniqueName: \"kubernetes.io/projected/357bc490-b170-466b-abf9-0ca8508cc257-kube-api-access-px4z4\") pod \"357bc490-b170-466b-abf9-0ca8508cc257\" (UID: \"357bc490-b170-466b-abf9-0ca8508cc257\") " Jul 31 19:34:06.960410 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:06.960287 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/357bc490-b170-466b-abf9-0ca8508cc257-tmp-dir\") pod \"357bc490-b170-466b-abf9-0ca8508cc257\" (UID: \"357bc490-b170-466b-abf9-0ca8508cc257\") " Jul 31 19:34:06.960410 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:06.960324 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/357bc490-b170-466b-abf9-0ca8508cc257-tls-certs\") pod \"357bc490-b170-466b-abf9-0ca8508cc257\" (UID: \"357bc490-b170-466b-abf9-0ca8508cc257\") " Jul 31 19:34:06.960643 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:06.960428 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357bc490-b170-466b-abf9-0ca8508cc257-home" (OuterVolumeSpecName: "home") pod "357bc490-b170-466b-abf9-0ca8508cc257" (UID: "357bc490-b170-466b-abf9-0ca8508cc257"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:34:06.960643 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:06.960442 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357bc490-b170-466b-abf9-0ca8508cc257-model-cache" (OuterVolumeSpecName: "model-cache") pod "357bc490-b170-466b-abf9-0ca8508cc257" (UID: "357bc490-b170-466b-abf9-0ca8508cc257"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:34:06.960643 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:06.960595 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357bc490-b170-466b-abf9-0ca8508cc257-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "357bc490-b170-466b-abf9-0ca8508cc257" (UID: "357bc490-b170-466b-abf9-0ca8508cc257"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:34:06.960797 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:06.960669 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/357bc490-b170-466b-abf9-0ca8508cc257-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:34:06.960797 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:06.960686 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/357bc490-b170-466b-abf9-0ca8508cc257-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:34:06.962518 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:06.962492 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357bc490-b170-466b-abf9-0ca8508cc257-dshm" (OuterVolumeSpecName: "dshm") pod "357bc490-b170-466b-abf9-0ca8508cc257" (UID: "357bc490-b170-466b-abf9-0ca8508cc257"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:34:06.962636 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:06.962519 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/357bc490-b170-466b-abf9-0ca8508cc257-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "357bc490-b170-466b-abf9-0ca8508cc257" (UID: "357bc490-b170-466b-abf9-0ca8508cc257"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 19:34:06.962636 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:06.962538 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/357bc490-b170-466b-abf9-0ca8508cc257-kube-api-access-px4z4" (OuterVolumeSpecName: "kube-api-access-px4z4") pod "357bc490-b170-466b-abf9-0ca8508cc257" (UID: "357bc490-b170-466b-abf9-0ca8508cc257"). InnerVolumeSpecName "kube-api-access-px4z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:34:07.062067 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:07.062031 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px4z4\" (UniqueName: \"kubernetes.io/projected/357bc490-b170-466b-abf9-0ca8508cc257-kube-api-access-px4z4\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:34:07.062067 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:07.062059 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/357bc490-b170-466b-abf9-0ca8508cc257-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:34:07.062067 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:07.062069 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/357bc490-b170-466b-abf9-0ca8508cc257-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:34:07.062067 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:07.062078 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/357bc490-b170-466b-abf9-0ca8508cc257-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:34:07.073214 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:07.073185 2565 generic.go:356] "Generic (PLEG): container finished" podID="357bc490-b170-466b-abf9-0ca8508cc257" containerID="da20e2eefff32c49fe934e534cfd67470df4a51ca1cf6af7f5aae529372e855a" exitCode=0 Jul 31 19:34:07.073369 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:07.073232 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8" Jul 31 19:34:07.073369 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:07.073273 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8" event={"ID":"357bc490-b170-466b-abf9-0ca8508cc257","Type":"ContainerDied","Data":"da20e2eefff32c49fe934e534cfd67470df4a51ca1cf6af7f5aae529372e855a"} Jul 31 19:34:07.073369 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:07.073311 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8" event={"ID":"357bc490-b170-466b-abf9-0ca8508cc257","Type":"ContainerDied","Data":"93ac1bb2f700f26e1507b959f6191508e19fd10cb60950f912d893d29901c13f"} Jul 31 19:34:07.073369 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:07.073330 2565 scope.go:117] "RemoveContainer" containerID="da20e2eefff32c49fe934e534cfd67470df4a51ca1cf6af7f5aae529372e855a" Jul 31 19:34:07.082314 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:07.082256 2565 scope.go:117] "RemoveContainer" containerID="da20e2eefff32c49fe934e534cfd67470df4a51ca1cf6af7f5aae529372e855a" Jul 31 19:34:07.082546 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:34:07.082529 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da20e2eefff32c49fe934e534cfd67470df4a51ca1cf6af7f5aae529372e855a\": container with ID starting with da20e2eefff32c49fe934e534cfd67470df4a51ca1cf6af7f5aae529372e855a not found: ID does not exist" containerID="da20e2eefff32c49fe934e534cfd67470df4a51ca1cf6af7f5aae529372e855a" Jul 31 19:34:07.082586 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:07.082555 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da20e2eefff32c49fe934e534cfd67470df4a51ca1cf6af7f5aae529372e855a"} err="failed to get container status \"da20e2eefff32c49fe934e534cfd67470df4a51ca1cf6af7f5aae529372e855a\": rpc error: code = NotFound desc = could not find container \"da20e2eefff32c49fe934e534cfd67470df4a51ca1cf6af7f5aae529372e855a\": container with ID starting with da20e2eefff32c49fe934e534cfd67470df4a51ca1cf6af7f5aae529372e855a not found: ID does not exist" Jul 31 19:34:07.096178 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:07.096150 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8"] Jul 31 19:34:07.101455 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:07.101431 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-fd0284/auth-test-model-fd0284-kserve-7d64949c47-2gbv8"] Jul 31 19:34:07.886524 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:07.886493 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="357bc490-b170-466b-abf9-0ca8508cc257" path="/var/lib/kubelet/pods/357bc490-b170-466b-abf9-0ca8508cc257/volumes" Jul 31 19:34:11.117099 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:11.117067 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x" Jul 31 19:34:11.117530 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:11.117115 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x" Jul 31 19:34:11.131040 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:11.131003 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x" Jul 31 19:34:12.117699 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:12.117670 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x" Jul 31 19:34:25.321280 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:25.321227 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:34:27.960139 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:27.960103 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:34:32.348871 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:34:32.348832 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:35:04.649556 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:35:04.649517 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:35:15.051345 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:35:15.051300 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:35:43.359447 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:35:43.359409 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:35:48.092080 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:35:48.092038 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x"] Jul 31 19:35:48.102947 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:35:48.102911 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:35:48.998021 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:35:48.997990 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:35:54.448280 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:35:54.448231 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:36:03.114512 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:03.114437 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x" podUID="888aeb55-6bb3-43ff-bce5-c70224451840" containerName="main" containerID="cri-o://cdd869c2e95d363866a6c0055052672e6dd6ffe3c9cf00925cd27d0b0424c1c8" gracePeriod=45 Jul 31 19:36:03.381356 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:03.381279 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x" Jul 31 19:36:03.389746 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:03.389714 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/888aeb55-6bb3-43ff-bce5-c70224451840-tmp-dir\") pod \"888aeb55-6bb3-43ff-bce5-c70224451840\" (UID: \"888aeb55-6bb3-43ff-bce5-c70224451840\") " Jul 31 19:36:03.389888 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:03.389780 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/888aeb55-6bb3-43ff-bce5-c70224451840-model-cache\") pod \"888aeb55-6bb3-43ff-bce5-c70224451840\" (UID: \"888aeb55-6bb3-43ff-bce5-c70224451840\") " Jul 31 19:36:03.389888 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:03.389816 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/888aeb55-6bb3-43ff-bce5-c70224451840-home\") pod \"888aeb55-6bb3-43ff-bce5-c70224451840\" (UID: \"888aeb55-6bb3-43ff-bce5-c70224451840\") " Jul 31 19:36:03.389888 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:03.389869 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/888aeb55-6bb3-43ff-bce5-c70224451840-dshm\") pod \"888aeb55-6bb3-43ff-bce5-c70224451840\" (UID: \"888aeb55-6bb3-43ff-bce5-c70224451840\") " Jul 31 19:36:03.390065 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:03.389921 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/888aeb55-6bb3-43ff-bce5-c70224451840-tls-certs\") pod \"888aeb55-6bb3-43ff-bce5-c70224451840\" (UID: \"888aeb55-6bb3-43ff-bce5-c70224451840\") " Jul 31 19:36:03.390065 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:03.389960 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvpqq\" (UniqueName: \"kubernetes.io/projected/888aeb55-6bb3-43ff-bce5-c70224451840-kube-api-access-nvpqq\") pod \"888aeb55-6bb3-43ff-bce5-c70224451840\" (UID: \"888aeb55-6bb3-43ff-bce5-c70224451840\") " Jul 31 19:36:03.390065 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:03.389971 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/888aeb55-6bb3-43ff-bce5-c70224451840-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "888aeb55-6bb3-43ff-bce5-c70224451840" (UID: "888aeb55-6bb3-43ff-bce5-c70224451840"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:36:03.390229 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:03.390060 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/888aeb55-6bb3-43ff-bce5-c70224451840-model-cache" (OuterVolumeSpecName: "model-cache") pod "888aeb55-6bb3-43ff-bce5-c70224451840" (UID: "888aeb55-6bb3-43ff-bce5-c70224451840"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:36:03.390229 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:03.390095 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/888aeb55-6bb3-43ff-bce5-c70224451840-home" (OuterVolumeSpecName: "home") pod "888aeb55-6bb3-43ff-bce5-c70224451840" (UID: "888aeb55-6bb3-43ff-bce5-c70224451840"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:36:03.390405 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:03.390385 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/888aeb55-6bb3-43ff-bce5-c70224451840-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:36:03.390468 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:03.390410 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/888aeb55-6bb3-43ff-bce5-c70224451840-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:36:03.390468 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:03.390424 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/888aeb55-6bb3-43ff-bce5-c70224451840-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:36:03.392006 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:03.391976 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/888aeb55-6bb3-43ff-bce5-c70224451840-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "888aeb55-6bb3-43ff-bce5-c70224451840" (UID: "888aeb55-6bb3-43ff-bce5-c70224451840"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 19:36:03.392006 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:03.392001 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/888aeb55-6bb3-43ff-bce5-c70224451840-kube-api-access-nvpqq" (OuterVolumeSpecName: "kube-api-access-nvpqq") pod "888aeb55-6bb3-43ff-bce5-c70224451840" (UID: "888aeb55-6bb3-43ff-bce5-c70224451840"). InnerVolumeSpecName "kube-api-access-nvpqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:36:03.392225 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:03.392209 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/888aeb55-6bb3-43ff-bce5-c70224451840-dshm" (OuterVolumeSpecName: "dshm") pod "888aeb55-6bb3-43ff-bce5-c70224451840" (UID: "888aeb55-6bb3-43ff-bce5-c70224451840"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:36:03.490913 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:03.490878 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/888aeb55-6bb3-43ff-bce5-c70224451840-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:36:03.490913 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:03.490910 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/888aeb55-6bb3-43ff-bce5-c70224451840-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:36:03.490913 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:03.490921 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvpqq\" (UniqueName: \"kubernetes.io/projected/888aeb55-6bb3-43ff-bce5-c70224451840-kube-api-access-nvpqq\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:36:03.676312 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:03.676203 2565 generic.go:356] "Generic (PLEG): container finished" podID="888aeb55-6bb3-43ff-bce5-c70224451840" containerID="cdd869c2e95d363866a6c0055052672e6dd6ffe3c9cf00925cd27d0b0424c1c8" exitCode=0 Jul 31 19:36:03.676312 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:03.676272 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x" Jul 31 19:36:03.676312 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:03.676276 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x" event={"ID":"888aeb55-6bb3-43ff-bce5-c70224451840","Type":"ContainerDied","Data":"cdd869c2e95d363866a6c0055052672e6dd6ffe3c9cf00925cd27d0b0424c1c8"} Jul 31 19:36:03.676547 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:03.676326 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x" event={"ID":"888aeb55-6bb3-43ff-bce5-c70224451840","Type":"ContainerDied","Data":"748b8e18c51cd405304b6b536e34d40fc57ee4ab1bf99cb5d9e2b2eafcdc32da"} Jul 31 19:36:03.676547 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:03.676346 2565 scope.go:117] "RemoveContainer" containerID="cdd869c2e95d363866a6c0055052672e6dd6ffe3c9cf00925cd27d0b0424c1c8" Jul 31 19:36:03.685581 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:03.685564 2565 scope.go:117] "RemoveContainer" containerID="cdd869c2e95d363866a6c0055052672e6dd6ffe3c9cf00925cd27d0b0424c1c8" Jul 31 19:36:03.685829 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:36:03.685811 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdd869c2e95d363866a6c0055052672e6dd6ffe3c9cf00925cd27d0b0424c1c8\": container with ID starting with cdd869c2e95d363866a6c0055052672e6dd6ffe3c9cf00925cd27d0b0424c1c8 not found: ID does not exist" containerID="cdd869c2e95d363866a6c0055052672e6dd6ffe3c9cf00925cd27d0b0424c1c8" Jul 31 19:36:03.685901 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:03.685835 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd869c2e95d363866a6c0055052672e6dd6ffe3c9cf00925cd27d0b0424c1c8"} err="failed to get container status \"cdd869c2e95d363866a6c0055052672e6dd6ffe3c9cf00925cd27d0b0424c1c8\": rpc error: code = NotFound desc = could not find container \"cdd869c2e95d363866a6c0055052672e6dd6ffe3c9cf00925cd27d0b0424c1c8\": container with ID starting with cdd869c2e95d363866a6c0055052672e6dd6ffe3c9cf00925cd27d0b0424c1c8 not found: ID does not exist" Jul 31 19:36:03.698213 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:03.698189 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x"] Jul 31 19:36:03.704382 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:03.704360 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-fd0284/sub-test-model-fd0284-kserve-7dd5d64b95-lm24x"] Jul 31 19:36:03.887806 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:03.887774 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="888aeb55-6bb3-43ff-bce5-c70224451840" path="/var/lib/kubelet/pods/888aeb55-6bb3-43ff-bce5-c70224451840/volumes" Jul 31 19:36:08.897632 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:08.897550 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk"] Jul 31 19:36:08.897992 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:08.897934 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="357bc490-b170-466b-abf9-0ca8508cc257" containerName="main" Jul 31 19:36:08.897992 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:08.897944 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="357bc490-b170-466b-abf9-0ca8508cc257" containerName="main" Jul 31 19:36:08.897992 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:08.897963 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="888aeb55-6bb3-43ff-bce5-c70224451840" containerName="main" Jul 31 19:36:08.897992 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:08.897969 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="888aeb55-6bb3-43ff-bce5-c70224451840" containerName="main" Jul 31 19:36:08.898120 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:08.898031 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="357bc490-b170-466b-abf9-0ca8508cc257" containerName="main" Jul 31 19:36:08.898120 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:08.898039 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="888aeb55-6bb3-43ff-bce5-c70224451840" containerName="main" Jul 31 19:36:08.902291 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:08.902269 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk" Jul 31 19:36:08.904374 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:08.904353 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-fd0284\"/\"kube-root-ca.crt\"" Jul 31 19:36:08.904935 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:08.904912 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-fd0284\"/\"openshift-service-ca.crt\"" Jul 31 19:36:08.905020 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:08.904940 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-fd0284\"/\"default-dockercfg-s5bzh\"" Jul 31 19:36:08.905020 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:08.904951 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-fd0284\"/\"rate-test-model-fd0284-kserve-self-signed-certs\"" Jul 31 19:36:08.910179 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:08.910153 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk"] Jul 31 19:36:08.924126 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:08.924084 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:36:08.942799 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:08.942769 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e683e01-2c9d-49f1-b14e-12ed905f0db8-tls-certs\") pod \"rate-test-model-fd0284-kserve-75bb547d47-twdgk\" (UID: \"3e683e01-2c9d-49f1-b14e-12ed905f0db8\") " pod="ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk" Jul 31 19:36:08.942931 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:08.942861 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e683e01-2c9d-49f1-b14e-12ed905f0db8-home\") pod \"rate-test-model-fd0284-kserve-75bb547d47-twdgk\" (UID: \"3e683e01-2c9d-49f1-b14e-12ed905f0db8\") " pod="ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk" Jul 31 19:36:08.942931 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:08.942914 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e683e01-2c9d-49f1-b14e-12ed905f0db8-dshm\") pod \"rate-test-model-fd0284-kserve-75bb547d47-twdgk\" (UID: \"3e683e01-2c9d-49f1-b14e-12ed905f0db8\") " pod="ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk" Jul 31 19:36:08.943027 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:08.942990 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e683e01-2c9d-49f1-b14e-12ed905f0db8-model-cache\") pod \"rate-test-model-fd0284-kserve-75bb547d47-twdgk\" (UID: \"3e683e01-2c9d-49f1-b14e-12ed905f0db8\") " pod="ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk" Jul 31 19:36:08.943027 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:08.943017 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzrrn\" (UniqueName: \"kubernetes.io/projected/3e683e01-2c9d-49f1-b14e-12ed905f0db8-kube-api-access-xzrrn\") pod \"rate-test-model-fd0284-kserve-75bb547d47-twdgk\" (UID: \"3e683e01-2c9d-49f1-b14e-12ed905f0db8\") " pod="ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk" Jul 31 19:36:08.943109 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:08.943081 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e683e01-2c9d-49f1-b14e-12ed905f0db8-tmp-dir\") pod \"rate-test-model-fd0284-kserve-75bb547d47-twdgk\" (UID: \"3e683e01-2c9d-49f1-b14e-12ed905f0db8\") " pod="ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk" Jul 31 19:36:09.044526 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:09.044485 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e683e01-2c9d-49f1-b14e-12ed905f0db8-home\") pod \"rate-test-model-fd0284-kserve-75bb547d47-twdgk\" (UID: \"3e683e01-2c9d-49f1-b14e-12ed905f0db8\") " pod="ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk" Jul 31 19:36:09.044526 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:09.044530 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e683e01-2c9d-49f1-b14e-12ed905f0db8-dshm\") pod \"rate-test-model-fd0284-kserve-75bb547d47-twdgk\" (UID: \"3e683e01-2c9d-49f1-b14e-12ed905f0db8\") " pod="ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk" Jul 31 19:36:09.044786 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:09.044602 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e683e01-2c9d-49f1-b14e-12ed905f0db8-model-cache\") pod \"rate-test-model-fd0284-kserve-75bb547d47-twdgk\" (UID: \"3e683e01-2c9d-49f1-b14e-12ed905f0db8\") " pod="ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk" Jul 31 19:36:09.044786 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:09.044639 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzrrn\" (UniqueName: \"kubernetes.io/projected/3e683e01-2c9d-49f1-b14e-12ed905f0db8-kube-api-access-xzrrn\") pod \"rate-test-model-fd0284-kserve-75bb547d47-twdgk\" (UID: \"3e683e01-2c9d-49f1-b14e-12ed905f0db8\") " pod="ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk" Jul 31 19:36:09.044786 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:09.044679 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e683e01-2c9d-49f1-b14e-12ed905f0db8-tmp-dir\") pod \"rate-test-model-fd0284-kserve-75bb547d47-twdgk\" (UID: \"3e683e01-2c9d-49f1-b14e-12ed905f0db8\") " pod="ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk" Jul 31 19:36:09.044786 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:09.044744 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e683e01-2c9d-49f1-b14e-12ed905f0db8-tls-certs\") pod \"rate-test-model-fd0284-kserve-75bb547d47-twdgk\" (UID: \"3e683e01-2c9d-49f1-b14e-12ed905f0db8\") " pod="ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk" Jul 31 19:36:09.044997 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:09.044915 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e683e01-2c9d-49f1-b14e-12ed905f0db8-home\") pod \"rate-test-model-fd0284-kserve-75bb547d47-twdgk\" (UID: \"3e683e01-2c9d-49f1-b14e-12ed905f0db8\") " pod="ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk" Jul 31 19:36:09.045056 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:09.045007 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e683e01-2c9d-49f1-b14e-12ed905f0db8-model-cache\") pod \"rate-test-model-fd0284-kserve-75bb547d47-twdgk\" (UID: \"3e683e01-2c9d-49f1-b14e-12ed905f0db8\") " pod="ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk" Jul 31 19:36:09.045112 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:09.045059 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e683e01-2c9d-49f1-b14e-12ed905f0db8-tmp-dir\") pod \"rate-test-model-fd0284-kserve-75bb547d47-twdgk\" (UID: \"3e683e01-2c9d-49f1-b14e-12ed905f0db8\") " pod="ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk" Jul 31 19:36:09.046764 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:09.046743 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e683e01-2c9d-49f1-b14e-12ed905f0db8-dshm\") pod \"rate-test-model-fd0284-kserve-75bb547d47-twdgk\" (UID: \"3e683e01-2c9d-49f1-b14e-12ed905f0db8\") " pod="ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk" Jul 31 19:36:09.047064 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:09.047046 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e683e01-2c9d-49f1-b14e-12ed905f0db8-tls-certs\") pod \"rate-test-model-fd0284-kserve-75bb547d47-twdgk\" (UID: \"3e683e01-2c9d-49f1-b14e-12ed905f0db8\") " pod="ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk" Jul 31 19:36:09.062711 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:09.062682 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzrrn\" (UniqueName: \"kubernetes.io/projected/3e683e01-2c9d-49f1-b14e-12ed905f0db8-kube-api-access-xzrrn\") pod \"rate-test-model-fd0284-kserve-75bb547d47-twdgk\" (UID: \"3e683e01-2c9d-49f1-b14e-12ed905f0db8\") " pod="ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk" Jul 31 19:36:09.213032 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:09.212941 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk" Jul 31 19:36:09.345531 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:09.345475 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk"] Jul 31 19:36:09.347359 ip-10-0-139-39 kubenswrapper[2565]: W0731 19:36:09.347332 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e683e01_2c9d_49f1_b14e_12ed905f0db8.slice/crio-61979bb6d6dac6cdc88a9c3d24abf92ca726d6b0ef36bdac3415957deb6839c7 WatchSource:0}: Error finding container 61979bb6d6dac6cdc88a9c3d24abf92ca726d6b0ef36bdac3415957deb6839c7: Status 404 returned error can't find the container with id 61979bb6d6dac6cdc88a9c3d24abf92ca726d6b0ef36bdac3415957deb6839c7 Jul 31 19:36:09.708700 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:09.708664 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk" event={"ID":"3e683e01-2c9d-49f1-b14e-12ed905f0db8","Type":"ContainerStarted","Data":"fe567978f684ed03694f0b7abbb55e6862208728b498147240b7c60f8d09a8b4"} Jul 31 19:36:09.708883 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:09.708708 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk" event={"ID":"3e683e01-2c9d-49f1-b14e-12ed905f0db8","Type":"ContainerStarted","Data":"61979bb6d6dac6cdc88a9c3d24abf92ca726d6b0ef36bdac3415957deb6839c7"} Jul 31 19:36:09.729193 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:09.729140 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk" podStartSLOduration=1.729125147 podStartE2EDuration="1.729125147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:36:09.726479944 +0000 UTC m=+3146.364469760" watchObservedRunningTime="2026-07-31 19:36:09.729125147 +0000 UTC m=+3146.367114925" Jul 31 19:36:11.263095 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:11.263062 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:36:19.213334 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:19.213298 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk" Jul 31 19:36:19.213814 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:19.213349 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk" Jul 31 19:36:19.225898 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:19.225869 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk" Jul 31 19:36:19.771162 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:19.771134 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk" Jul 31 19:36:27.399328 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:27.399271 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:36:29.048008 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:36:29.047978 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:37:36.858570 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:37:36.858488 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:37:46.058466 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:37:46.058431 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:38:01.055298 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:01.055259 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:38:06.615423 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:06.615380 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk"] Jul 31 19:38:06.654114 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:06.654077 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:38:07.487058 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:07.487021 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:38:09.653790 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:09.653755 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:38:12.345524 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:12.345490 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:38:14.631046 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:14.631006 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7593-139919266618512"] Jul 31 19:38:14.634583 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:14.634565 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-139919266618512" Jul 31 19:38:14.636417 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:14.636390 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4rlcb\"" Jul 31 19:38:14.642026 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:14.641836 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7593-139919266618512"] Jul 31 19:38:14.735633 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:14.735595 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lls5\" (UniqueName: \"kubernetes.io/projected/3708b476-6630-40fc-a659-322c252d96d9-kube-api-access-5lls5\") pod \"test-curl-7593-139919266618512\" (UID: \"3708b476-6630-40fc-a659-322c252d96d9\") " pod="opendatahub/test-curl-7593-139919266618512" Jul 31 19:38:14.837073 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:14.837036 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lls5\" (UniqueName: \"kubernetes.io/projected/3708b476-6630-40fc-a659-322c252d96d9-kube-api-access-5lls5\") pod \"test-curl-7593-139919266618512\" (UID: \"3708b476-6630-40fc-a659-322c252d96d9\") " pod="opendatahub/test-curl-7593-139919266618512" Jul 31 19:38:14.847700 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:14.847666 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lls5\" (UniqueName: \"kubernetes.io/projected/3708b476-6630-40fc-a659-322c252d96d9-kube-api-access-5lls5\") pod \"test-curl-7593-139919266618512\" (UID: \"3708b476-6630-40fc-a659-322c252d96d9\") " pod="opendatahub/test-curl-7593-139919266618512" Jul 31 19:38:14.945222 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:14.945143 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-139919266618512" Jul 31 19:38:15.088328 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:15.088300 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7593-139919266618512"] Jul 31 19:38:15.091044 ip-10-0-139-39 kubenswrapper[2565]: W0731 19:38:15.091016 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3708b476_6630_40fc_a659_322c252d96d9.slice/crio-56a4e2b9f60e21230f9a4580feb08fc9323ea308e27c47feceb6ace3d10bf137 WatchSource:0}: Error finding container 56a4e2b9f60e21230f9a4580feb08fc9323ea308e27c47feceb6ace3d10bf137: Status 404 returned error can't find the container with id 56a4e2b9f60e21230f9a4580feb08fc9323ea308e27c47feceb6ace3d10bf137 Jul 31 19:38:15.092482 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:15.092462 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 19:38:15.331184 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:15.331146 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7593-139919266618512" event={"ID":"3708b476-6630-40fc-a659-322c252d96d9","Type":"ContainerStarted","Data":"56a4e2b9f60e21230f9a4580feb08fc9323ea308e27c47feceb6ace3d10bf137"} Jul 31 19:38:16.337853 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:16.337817 2565 generic.go:356] "Generic (PLEG): container finished" podID="3708b476-6630-40fc-a659-322c252d96d9" containerID="a1f71e23fa451662a449b59989ee3c03b40291ac26b79d89bac42bb8ffa01e81" exitCode=0 Jul 31 19:38:16.338257 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:16.337890 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7593-139919266618512" event={"ID":"3708b476-6630-40fc-a659-322c252d96d9","Type":"ContainerDied","Data":"a1f71e23fa451662a449b59989ee3c03b40291ac26b79d89bac42bb8ffa01e81"} Jul 31 19:38:17.478685 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:17.478660 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-139919266618512" Jul 31 19:38:17.500255 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:17.500204 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7593-139919266618512_3708b476-6630-40fc-a659-322c252d96d9/test-curl-7593-139919266618512/0.log" Jul 31 19:38:17.519652 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:17.519619 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7593-139919266618512"] Jul 31 19:38:17.524673 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:17.524649 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7593-139919266618512"] Jul 31 19:38:17.566574 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:17.566542 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lls5\" (UniqueName: \"kubernetes.io/projected/3708b476-6630-40fc-a659-322c252d96d9-kube-api-access-5lls5\") pod \"3708b476-6630-40fc-a659-322c252d96d9\" (UID: \"3708b476-6630-40fc-a659-322c252d96d9\") " Jul 31 19:38:17.568726 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:17.568701 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3708b476-6630-40fc-a659-322c252d96d9-kube-api-access-5lls5" (OuterVolumeSpecName: "kube-api-access-5lls5") pod "3708b476-6630-40fc-a659-322c252d96d9" (UID: "3708b476-6630-40fc-a659-322c252d96d9"). InnerVolumeSpecName "kube-api-access-5lls5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:38:17.667665 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:17.667577 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lls5\" (UniqueName: \"kubernetes.io/projected/3708b476-6630-40fc-a659-322c252d96d9-kube-api-access-5lls5\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:38:17.886836 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:17.886798 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3708b476-6630-40fc-a659-322c252d96d9" path="/var/lib/kubelet/pods/3708b476-6630-40fc-a659-322c252d96d9/volumes" Jul 31 19:38:18.349454 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:18.349424 2565 scope.go:117] "RemoveContainer" containerID="a1f71e23fa451662a449b59989ee3c03b40291ac26b79d89bac42bb8ffa01e81" Jul 31 19:38:18.349605 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:18.349426 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-139919266618512" Jul 31 19:38:20.644406 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:20.644362 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7593-139919266684976"] Jul 31 19:38:20.644801 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:20.644773 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3708b476-6630-40fc-a659-322c252d96d9" containerName="test-curl-7593-139919266618512" Jul 31 19:38:20.644801 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:20.644785 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3708b476-6630-40fc-a659-322c252d96d9" containerName="test-curl-7593-139919266618512" Jul 31 19:38:20.644872 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:20.644854 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3708b476-6630-40fc-a659-322c252d96d9" containerName="test-curl-7593-139919266618512" Jul 31 19:38:20.649427 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:20.649399 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-139919266684976" Jul 31 19:38:20.651298 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:20.651280 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4rlcb\"" Jul 31 19:38:20.655943 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:20.655922 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7593-139919266684976"] Jul 31 19:38:20.698763 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:20.698732 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95lzc\" (UniqueName: \"kubernetes.io/projected/fb5f89ac-c909-4163-b10d-9fbab7b6bba0-kube-api-access-95lzc\") pod \"test-curl-7593-139919266684976\" (UID: \"fb5f89ac-c909-4163-b10d-9fbab7b6bba0\") " pod="opendatahub/test-curl-7593-139919266684976" Jul 31 19:38:20.799841 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:20.799803 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95lzc\" (UniqueName: \"kubernetes.io/projected/fb5f89ac-c909-4163-b10d-9fbab7b6bba0-kube-api-access-95lzc\") pod \"test-curl-7593-139919266684976\" (UID: \"fb5f89ac-c909-4163-b10d-9fbab7b6bba0\") " pod="opendatahub/test-curl-7593-139919266684976" Jul 31 19:38:20.808751 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:20.808719 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95lzc\" (UniqueName: \"kubernetes.io/projected/fb5f89ac-c909-4163-b10d-9fbab7b6bba0-kube-api-access-95lzc\") pod \"test-curl-7593-139919266684976\" (UID: \"fb5f89ac-c909-4163-b10d-9fbab7b6bba0\") " pod="opendatahub/test-curl-7593-139919266684976" Jul 31 19:38:20.959699 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:20.959609 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-139919266684976" Jul 31 19:38:21.099336 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:21.099307 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7593-139919266684976"] Jul 31 19:38:21.101371 ip-10-0-139-39 kubenswrapper[2565]: W0731 19:38:21.101345 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb5f89ac_c909_4163_b10d_9fbab7b6bba0.slice/crio-2024e0ddd2dac5122a9ad0a40a3421fed891087f0e046f757906892b736d834e WatchSource:0}: Error finding container 2024e0ddd2dac5122a9ad0a40a3421fed891087f0e046f757906892b736d834e: Status 404 returned error can't find the container with id 2024e0ddd2dac5122a9ad0a40a3421fed891087f0e046f757906892b736d834e Jul 31 19:38:21.367119 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:21.367081 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7593-139919266684976" event={"ID":"fb5f89ac-c909-4163-b10d-9fbab7b6bba0","Type":"ContainerStarted","Data":"2024e0ddd2dac5122a9ad0a40a3421fed891087f0e046f757906892b736d834e"} Jul 31 19:38:21.634467 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:21.634368 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk" podUID="3e683e01-2c9d-49f1-b14e-12ed905f0db8" containerName="main" containerID="cri-o://fe567978f684ed03694f0b7abbb55e6862208728b498147240b7c60f8d09a8b4" gracePeriod=45 Jul 31 19:38:21.892250 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:21.892176 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk" Jul 31 19:38:22.013038 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:22.012997 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e683e01-2c9d-49f1-b14e-12ed905f0db8-model-cache\") pod \"3e683e01-2c9d-49f1-b14e-12ed905f0db8\" (UID: \"3e683e01-2c9d-49f1-b14e-12ed905f0db8\") " Jul 31 19:38:22.013038 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:22.013041 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e683e01-2c9d-49f1-b14e-12ed905f0db8-dshm\") pod \"3e683e01-2c9d-49f1-b14e-12ed905f0db8\" (UID: \"3e683e01-2c9d-49f1-b14e-12ed905f0db8\") " Jul 31 19:38:22.013335 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:22.013067 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e683e01-2c9d-49f1-b14e-12ed905f0db8-tmp-dir\") pod \"3e683e01-2c9d-49f1-b14e-12ed905f0db8\" (UID: \"3e683e01-2c9d-49f1-b14e-12ed905f0db8\") " Jul 31 19:38:22.013335 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:22.013135 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e683e01-2c9d-49f1-b14e-12ed905f0db8-home\") pod \"3e683e01-2c9d-49f1-b14e-12ed905f0db8\" (UID: \"3e683e01-2c9d-49f1-b14e-12ed905f0db8\") " Jul 31 19:38:22.013335 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:22.013185 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzrrn\" (UniqueName: \"kubernetes.io/projected/3e683e01-2c9d-49f1-b14e-12ed905f0db8-kube-api-access-xzrrn\") pod \"3e683e01-2c9d-49f1-b14e-12ed905f0db8\" (UID: \"3e683e01-2c9d-49f1-b14e-12ed905f0db8\") " Jul 31 19:38:22.013335 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:22.013271 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e683e01-2c9d-49f1-b14e-12ed905f0db8-tls-certs\") pod \"3e683e01-2c9d-49f1-b14e-12ed905f0db8\" (UID: \"3e683e01-2c9d-49f1-b14e-12ed905f0db8\") " Jul 31 19:38:22.013536 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:22.013363 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e683e01-2c9d-49f1-b14e-12ed905f0db8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3e683e01-2c9d-49f1-b14e-12ed905f0db8" (UID: "3e683e01-2c9d-49f1-b14e-12ed905f0db8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:38:22.013536 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:22.013377 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e683e01-2c9d-49f1-b14e-12ed905f0db8-model-cache" (OuterVolumeSpecName: "model-cache") pod "3e683e01-2c9d-49f1-b14e-12ed905f0db8" (UID: "3e683e01-2c9d-49f1-b14e-12ed905f0db8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:38:22.013536 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:22.013445 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e683e01-2c9d-49f1-b14e-12ed905f0db8-home" (OuterVolumeSpecName: "home") pod "3e683e01-2c9d-49f1-b14e-12ed905f0db8" (UID: "3e683e01-2c9d-49f1-b14e-12ed905f0db8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:38:22.013772 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:22.013752 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e683e01-2c9d-49f1-b14e-12ed905f0db8-model-cache\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:38:22.013854 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:22.013785 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e683e01-2c9d-49f1-b14e-12ed905f0db8-tmp-dir\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:38:22.013854 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:22.013800 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e683e01-2c9d-49f1-b14e-12ed905f0db8-home\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:38:22.015355 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:22.015334 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e683e01-2c9d-49f1-b14e-12ed905f0db8-kube-api-access-xzrrn" (OuterVolumeSpecName: "kube-api-access-xzrrn") pod "3e683e01-2c9d-49f1-b14e-12ed905f0db8" (UID: "3e683e01-2c9d-49f1-b14e-12ed905f0db8"). InnerVolumeSpecName "kube-api-access-xzrrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:38:22.015803 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:22.015786 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e683e01-2c9d-49f1-b14e-12ed905f0db8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3e683e01-2c9d-49f1-b14e-12ed905f0db8" (UID: "3e683e01-2c9d-49f1-b14e-12ed905f0db8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 19:38:22.015803 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:22.015791 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e683e01-2c9d-49f1-b14e-12ed905f0db8-dshm" (OuterVolumeSpecName: "dshm") pod "3e683e01-2c9d-49f1-b14e-12ed905f0db8" (UID: "3e683e01-2c9d-49f1-b14e-12ed905f0db8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:38:22.114411 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:22.114373 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e683e01-2c9d-49f1-b14e-12ed905f0db8-tls-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:38:22.114411 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:22.114402 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e683e01-2c9d-49f1-b14e-12ed905f0db8-dshm\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:38:22.114411 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:22.114412 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzrrn\" (UniqueName: \"kubernetes.io/projected/3e683e01-2c9d-49f1-b14e-12ed905f0db8-kube-api-access-xzrrn\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:38:22.374370 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:22.374333 2565 generic.go:356] "Generic (PLEG): container finished" podID="fb5f89ac-c909-4163-b10d-9fbab7b6bba0" containerID="c74fb8bc603fde4e1f0c32a1675a98bd7030abe0efcb88036295ba5ea94a77a3" exitCode=0 Jul 31 19:38:22.374532 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:22.374418 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7593-139919266684976" event={"ID":"fb5f89ac-c909-4163-b10d-9fbab7b6bba0","Type":"ContainerDied","Data":"c74fb8bc603fde4e1f0c32a1675a98bd7030abe0efcb88036295ba5ea94a77a3"} Jul 31 19:38:22.375576 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:22.375553 2565 generic.go:356] "Generic (PLEG): container finished" podID="3e683e01-2c9d-49f1-b14e-12ed905f0db8" containerID="fe567978f684ed03694f0b7abbb55e6862208728b498147240b7c60f8d09a8b4" exitCode=0 Jul 31 19:38:22.375690 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:22.375587 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk" Jul 31 19:38:22.375690 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:22.375595 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk" event={"ID":"3e683e01-2c9d-49f1-b14e-12ed905f0db8","Type":"ContainerDied","Data":"fe567978f684ed03694f0b7abbb55e6862208728b498147240b7c60f8d09a8b4"} Jul 31 19:38:22.375690 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:22.375618 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk" event={"ID":"3e683e01-2c9d-49f1-b14e-12ed905f0db8","Type":"ContainerDied","Data":"61979bb6d6dac6cdc88a9c3d24abf92ca726d6b0ef36bdac3415957deb6839c7"} Jul 31 19:38:22.375690 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:22.375633 2565 scope.go:117] "RemoveContainer" containerID="fe567978f684ed03694f0b7abbb55e6862208728b498147240b7c60f8d09a8b4" Jul 31 19:38:22.384509 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:22.384491 2565 scope.go:117] "RemoveContainer" containerID="fe567978f684ed03694f0b7abbb55e6862208728b498147240b7c60f8d09a8b4" Jul 31 19:38:22.384783 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:38:22.384764 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe567978f684ed03694f0b7abbb55e6862208728b498147240b7c60f8d09a8b4\": container with ID starting with fe567978f684ed03694f0b7abbb55e6862208728b498147240b7c60f8d09a8b4 not found: ID does not exist" containerID="fe567978f684ed03694f0b7abbb55e6862208728b498147240b7c60f8d09a8b4" Jul 31 19:38:22.384834 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:22.384790 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe567978f684ed03694f0b7abbb55e6862208728b498147240b7c60f8d09a8b4"} err="failed to get container status \"fe567978f684ed03694f0b7abbb55e6862208728b498147240b7c60f8d09a8b4\": rpc error: code = NotFound desc = could not find container \"fe567978f684ed03694f0b7abbb55e6862208728b498147240b7c60f8d09a8b4\": container with ID starting with fe567978f684ed03694f0b7abbb55e6862208728b498147240b7c60f8d09a8b4 not found: ID does not exist" Jul 31 19:38:22.401058 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:22.401029 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk"] Jul 31 19:38:22.405689 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:22.405657 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-fd0284/rate-test-model-fd0284-kserve-75bb547d47-twdgk"] Jul 31 19:38:23.520530 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:23.520500 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-139919266684976" Jul 31 19:38:23.541546 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:23.541513 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7593-139919266684976_fb5f89ac-c909-4163-b10d-9fbab7b6bba0/test-curl-7593-139919266684976/0.log" Jul 31 19:38:23.562574 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:23.562543 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7593-139919266684976"] Jul 31 19:38:23.566486 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:23.566462 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7593-139919266684976"] Jul 31 19:38:23.628498 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:23.628465 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95lzc\" (UniqueName: \"kubernetes.io/projected/fb5f89ac-c909-4163-b10d-9fbab7b6bba0-kube-api-access-95lzc\") pod \"fb5f89ac-c909-4163-b10d-9fbab7b6bba0\" (UID: \"fb5f89ac-c909-4163-b10d-9fbab7b6bba0\") " Jul 31 19:38:23.630543 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:23.630513 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb5f89ac-c909-4163-b10d-9fbab7b6bba0-kube-api-access-95lzc" (OuterVolumeSpecName: "kube-api-access-95lzc") pod "fb5f89ac-c909-4163-b10d-9fbab7b6bba0" (UID: "fb5f89ac-c909-4163-b10d-9fbab7b6bba0"). InnerVolumeSpecName "kube-api-access-95lzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:38:23.687593 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:23.687558 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7593-139919266684976"] Jul 31 19:38:23.687965 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:23.687952 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e683e01-2c9d-49f1-b14e-12ed905f0db8" containerName="main" Jul 31 19:38:23.688011 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:23.687967 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e683e01-2c9d-49f1-b14e-12ed905f0db8" containerName="main" Jul 31 19:38:23.688011 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:23.687982 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb5f89ac-c909-4163-b10d-9fbab7b6bba0" containerName="test-curl-7593-139919266684976" Jul 31 19:38:23.688011 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:23.687988 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb5f89ac-c909-4163-b10d-9fbab7b6bba0" containerName="test-curl-7593-139919266684976" Jul 31 19:38:23.688103 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:23.688067 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb5f89ac-c909-4163-b10d-9fbab7b6bba0" containerName="test-curl-7593-139919266684976" Jul 31 19:38:23.688103 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:23.688076 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e683e01-2c9d-49f1-b14e-12ed905f0db8" containerName="main" Jul 31 19:38:23.691190 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:23.691171 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-139919266684976" Jul 31 19:38:23.695574 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:23.695541 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7593-139919266684976"] Jul 31 19:38:23.729376 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:23.729346 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vmn5\" (UniqueName: \"kubernetes.io/projected/1636ebe7-d1df-4052-949a-10f8f61509bd-kube-api-access-6vmn5\") pod \"test-curl-7593-139919266684976\" (UID: \"1636ebe7-d1df-4052-949a-10f8f61509bd\") " pod="opendatahub/test-curl-7593-139919266684976" Jul 31 19:38:23.729558 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:23.729540 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95lzc\" (UniqueName: \"kubernetes.io/projected/fb5f89ac-c909-4163-b10d-9fbab7b6bba0-kube-api-access-95lzc\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:38:23.830827 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:23.830786 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vmn5\" (UniqueName: \"kubernetes.io/projected/1636ebe7-d1df-4052-949a-10f8f61509bd-kube-api-access-6vmn5\") pod \"test-curl-7593-139919266684976\" (UID: \"1636ebe7-d1df-4052-949a-10f8f61509bd\") " pod="opendatahub/test-curl-7593-139919266684976" Jul 31 19:38:23.839130 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:23.839096 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vmn5\" (UniqueName: \"kubernetes.io/projected/1636ebe7-d1df-4052-949a-10f8f61509bd-kube-api-access-6vmn5\") pod \"test-curl-7593-139919266684976\" (UID: \"1636ebe7-d1df-4052-949a-10f8f61509bd\") " pod="opendatahub/test-curl-7593-139919266684976" Jul 31 19:38:23.888698 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:23.888598 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e683e01-2c9d-49f1-b14e-12ed905f0db8" path="/var/lib/kubelet/pods/3e683e01-2c9d-49f1-b14e-12ed905f0db8/volumes" Jul 31 19:38:23.889073 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:23.889052 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb5f89ac-c909-4163-b10d-9fbab7b6bba0" path="/var/lib/kubelet/pods/fb5f89ac-c909-4163-b10d-9fbab7b6bba0/volumes" Jul 31 19:38:24.002763 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:24.002733 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-139919266684976" Jul 31 19:38:24.142924 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:24.142864 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7593-139919266684976"] Jul 31 19:38:24.145522 ip-10-0-139-39 kubenswrapper[2565]: W0731 19:38:24.145496 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1636ebe7_d1df_4052_949a_10f8f61509bd.slice/crio-92c02335e9807b270bd613a104c60517d922a149261f0223158b45fd8bb148fb WatchSource:0}: Error finding container 92c02335e9807b270bd613a104c60517d922a149261f0223158b45fd8bb148fb: Status 404 returned error can't find the container with id 92c02335e9807b270bd613a104c60517d922a149261f0223158b45fd8bb148fb Jul 31 19:38:24.387047 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:24.387016 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-139919266684976" Jul 31 19:38:24.387222 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:24.387063 2565 scope.go:117] "RemoveContainer" containerID="c74fb8bc603fde4e1f0c32a1675a98bd7030abe0efcb88036295ba5ea94a77a3" Jul 31 19:38:24.388398 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:24.388329 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7593-139919266684976" event={"ID":"1636ebe7-d1df-4052-949a-10f8f61509bd","Type":"ContainerStarted","Data":"92c02335e9807b270bd613a104c60517d922a149261f0223158b45fd8bb148fb"} Jul 31 19:38:25.396480 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:25.396448 2565 generic.go:356] "Generic (PLEG): container finished" podID="1636ebe7-d1df-4052-949a-10f8f61509bd" containerID="8215fae230c3b3b9448baddb1312e32a744809e55558d938fe732fa01d422a0a" exitCode=0 Jul 31 19:38:25.396880 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:25.396526 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7593-139919266684976" event={"ID":"1636ebe7-d1df-4052-949a-10f8f61509bd","Type":"ContainerDied","Data":"8215fae230c3b3b9448baddb1312e32a744809e55558d938fe732fa01d422a0a"} Jul 31 19:38:26.541824 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:26.541799 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-139919266684976" Jul 31 19:38:26.560969 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:26.560944 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7593-139919266684976_1636ebe7-d1df-4052-949a-10f8f61509bd/test-curl-7593-139919266684976/0.log" Jul 31 19:38:26.586670 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:26.586640 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7593-139919266684976"] Jul 31 19:38:26.590267 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:26.590234 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7593-139919266684976"] Jul 31 19:38:26.659880 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:26.659790 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vmn5\" (UniqueName: \"kubernetes.io/projected/1636ebe7-d1df-4052-949a-10f8f61509bd-kube-api-access-6vmn5\") pod \"1636ebe7-d1df-4052-949a-10f8f61509bd\" (UID: \"1636ebe7-d1df-4052-949a-10f8f61509bd\") " Jul 31 19:38:26.661918 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:26.661885 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1636ebe7-d1df-4052-949a-10f8f61509bd-kube-api-access-6vmn5" (OuterVolumeSpecName: "kube-api-access-6vmn5") pod "1636ebe7-d1df-4052-949a-10f8f61509bd" (UID: "1636ebe7-d1df-4052-949a-10f8f61509bd"). InnerVolumeSpecName "kube-api-access-6vmn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:38:26.761569 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:26.761529 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vmn5\" (UniqueName: \"kubernetes.io/projected/1636ebe7-d1df-4052-949a-10f8f61509bd-kube-api-access-6vmn5\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:38:27.091528 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.091490 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:38:27.371173 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.371076 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk"] Jul 31 19:38:27.371649 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.371616 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1636ebe7-d1df-4052-949a-10f8f61509bd" containerName="test-curl-7593-139919266684976" Jul 31 19:38:27.371649 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.371636 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1636ebe7-d1df-4052-949a-10f8f61509bd" containerName="test-curl-7593-139919266684976" Jul 31 19:38:27.371856 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.371752 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1636ebe7-d1df-4052-949a-10f8f61509bd" containerName="test-curl-7593-139919266684976" Jul 31 19:38:27.374977 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.374950 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:27.376876 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.376855 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-0368ff-openshift-default-dockercfg-6xrft\"" Jul 31 19:38:27.385770 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.385745 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk"] Jul 31 19:38:27.398101 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.398067 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:38:27.410293 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.410266 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-139919266684976" Jul 31 19:38:27.410466 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.410271 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92c02335e9807b270bd613a104c60517d922a149261f0223158b45fd8bb148fb" Jul 31 19:38:27.422109 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:38:27.422077 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7593-139919266684976\" is forbidden: User \"system:node:ip-10-0-139-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-39.ec2.internal' and this object" podUID="1636ebe7-d1df-4052-949a-10f8f61509bd" pod="opendatahub/test-curl-7593-139919266684976" Jul 31 19:38:27.469018 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.468986 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54fff827-a0b0-4999-9f01-7d438d3270f4-istiod-ca-cert\") pod \"e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:27.469018 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.469019 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vksq\" (UniqueName: \"kubernetes.io/projected/54fff827-a0b0-4999-9f01-7d438d3270f4-kube-api-access-4vksq\") pod \"e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:27.469226 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.469067 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54fff827-a0b0-4999-9f01-7d438d3270f4-istio-envoy\") pod \"e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:27.469226 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.469093 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fff827-a0b0-4999-9f01-7d438d3270f4-workload-socket\") pod \"e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:27.469226 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.469109 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54fff827-a0b0-4999-9f01-7d438d3270f4-istio-token\") pod \"e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:27.469226 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.469135 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54fff827-a0b0-4999-9f01-7d438d3270f4-workload-certs\") pod \"e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:27.469226 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.469184 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fff827-a0b0-4999-9f01-7d438d3270f4-credential-socket\") pod \"e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:27.469226 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.469206 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54fff827-a0b0-4999-9f01-7d438d3270f4-istio-data\") pod \"e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:27.469472 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.469295 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54fff827-a0b0-4999-9f01-7d438d3270f4-istio-podinfo\") pod \"e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:27.499497 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.499463 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:38:27.570901 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.570861 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fff827-a0b0-4999-9f01-7d438d3270f4-credential-socket\") pod \"e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:27.571419 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.570923 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54fff827-a0b0-4999-9f01-7d438d3270f4-istio-data\") pod \"e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:27.571419 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.570999 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54fff827-a0b0-4999-9f01-7d438d3270f4-istio-podinfo\") pod \"e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:27.571419 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.571038 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54fff827-a0b0-4999-9f01-7d438d3270f4-istiod-ca-cert\") pod \"e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:27.571419 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.571064 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vksq\" (UniqueName: \"kubernetes.io/projected/54fff827-a0b0-4999-9f01-7d438d3270f4-kube-api-access-4vksq\") pod \"e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:27.571419 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.571137 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54fff827-a0b0-4999-9f01-7d438d3270f4-istio-envoy\") pod \"e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:27.571419 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.571175 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fff827-a0b0-4999-9f01-7d438d3270f4-workload-socket\") pod \"e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:27.571419 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.571196 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54fff827-a0b0-4999-9f01-7d438d3270f4-istio-token\") pod \"e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:27.571419 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.571268 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54fff827-a0b0-4999-9f01-7d438d3270f4-workload-certs\") pod \"e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:27.571419 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.571386 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fff827-a0b0-4999-9f01-7d438d3270f4-credential-socket\") pod \"e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:27.571935 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.571663 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54fff827-a0b0-4999-9f01-7d438d3270f4-workload-certs\") pod \"e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:27.572021 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.571994 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54fff827-a0b0-4999-9f01-7d438d3270f4-istio-data\") pod \"e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:27.572760 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.572729 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fff827-a0b0-4999-9f01-7d438d3270f4-workload-socket\") pod \"e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:27.588989 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.580814 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54fff827-a0b0-4999-9f01-7d438d3270f4-istiod-ca-cert\") pod \"e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:27.588989 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.583331 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54fff827-a0b0-4999-9f01-7d438d3270f4-istio-envoy\") pod \"e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:27.588989 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.583360 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vksq\" (UniqueName: \"kubernetes.io/projected/54fff827-a0b0-4999-9f01-7d438d3270f4-kube-api-access-4vksq\") pod \"e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:27.588989 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.588865 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54fff827-a0b0-4999-9f01-7d438d3270f4-istio-podinfo\") pod \"e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:27.588989 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.588871 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54fff827-a0b0-4999-9f01-7d438d3270f4-istio-token\") pod \"e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:27.688530 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.688425 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:27.834856 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.834817 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk"] Jul 31 19:38:27.835195 ip-10-0-139-39 kubenswrapper[2565]: W0731 19:38:27.835166 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54fff827_a0b0_4999_9f01_7d438d3270f4.slice/crio-ccb4dce655a71e3a05afdf2f8accfd25f65a39a3c3806beec4b187ffb3e19b69 WatchSource:0}: Error finding container ccb4dce655a71e3a05afdf2f8accfd25f65a39a3c3806beec4b187ffb3e19b69: Status 404 returned error can't find the container with id ccb4dce655a71e3a05afdf2f8accfd25f65a39a3c3806beec4b187ffb3e19b69 Jul 31 19:38:27.837601 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.837561 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:38:27.837665 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.837649 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:38:27.837703 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.837692 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:38:27.888675 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:27.888644 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1636ebe7-d1df-4052-949a-10f8f61509bd" path="/var/lib/kubelet/pods/1636ebe7-d1df-4052-949a-10f8f61509bd/volumes" Jul 31 19:38:28.417225 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:28.417189 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" event={"ID":"54fff827-a0b0-4999-9f01-7d438d3270f4","Type":"ContainerStarted","Data":"6045fa43c551954a9d7afb0596d1d3cac211f9139e4b2f18832370dfcdc53447"} Jul 31 19:38:28.417225 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:28.417253 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" event={"ID":"54fff827-a0b0-4999-9f01-7d438d3270f4","Type":"ContainerStarted","Data":"ccb4dce655a71e3a05afdf2f8accfd25f65a39a3c3806beec4b187ffb3e19b69"} Jul 31 19:38:28.438626 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:28.438575 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" podStartSLOduration=1.438559611 podStartE2EDuration="1.438559611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:38:28.437002531 +0000 UTC m=+3285.074992331" watchObservedRunningTime="2026-07-31 19:38:28.438559611 +0000 UTC m=+3285.076549412" Jul 31 19:38:28.689353 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:28.689273 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:28.694121 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:28.694094 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:29.423104 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:29.423075 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:29.424006 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:29.423988 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:30.370980 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:30.370943 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:38:30.464166 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:30.464123 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:38:31.576148 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:31.576090 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:38:32.063377 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:32.063338 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:38:32.166757 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:32.166714 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:38:32.267732 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:32.267681 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:38:34.558772 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:34.558719 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:38:44.424324 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:44.424288 2565 patch_prober.go:28] interesting pod/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Jul 31 19:38:44.424732 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:44.424384 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" podUID="54fff827-a0b0-4999-9f01-7d438d3270f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Jul 31 19:38:44.504707 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:44.504678 2565 generic.go:356] "Generic (PLEG): container finished" podID="54fff827-a0b0-4999-9f01-7d438d3270f4" containerID="6045fa43c551954a9d7afb0596d1d3cac211f9139e4b2f18832370dfcdc53447" exitCode=0 Jul 31 19:38:44.504872 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:44.504725 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" event={"ID":"54fff827-a0b0-4999-9f01-7d438d3270f4","Type":"ContainerDied","Data":"6045fa43c551954a9d7afb0596d1d3cac211f9139e4b2f18832370dfcdc53447"} Jul 31 19:38:44.505163 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:44.505145 2565 scope.go:117] "RemoveContainer" containerID="6045fa43c551954a9d7afb0596d1d3cac211f9139e4b2f18832370dfcdc53447" Jul 31 19:38:44.506200 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:44.506171 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:38:44.506293 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:44.506277 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:38:44.506345 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:44.506314 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:38:45.511970 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:45.511931 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" event={"ID":"54fff827-a0b0-4999-9f01-7d438d3270f4","Type":"ContainerStarted","Data":"da619dfa18a07cd0f904bb2cb3d92b5c11c527ddfcea49b6764165f373812bf8"} Jul 31 19:38:45.689135 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:45.689098 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:45.689375 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:45.689348 2565 patch_prober.go:28] interesting pod/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Jul 31 19:38:45.689458 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:45.689424 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" podUID="54fff827-a0b0-4999-9f01-7d438d3270f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Jul 31 19:38:46.688962 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:46.688925 2565 patch_prober.go:28] interesting pod/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Jul 31 19:38:46.689385 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:46.688988 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" podUID="54fff827-a0b0-4999-9f01-7d438d3270f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Jul 31 19:38:47.689001 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:47.688962 2565 patch_prober.go:28] interesting pod/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Jul 31 19:38:47.689416 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:47.689022 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" podUID="54fff827-a0b0-4999-9f01-7d438d3270f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Jul 31 19:38:48.689722 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:48.689683 2565 patch_prober.go:28] interesting pod/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Jul 31 19:38:48.690126 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:48.689743 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" podUID="54fff827-a0b0-4999-9f01-7d438d3270f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Jul 31 19:38:49.689505 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:49.689465 2565 patch_prober.go:28] interesting pod/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Jul 31 19:38:49.689699 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:49.689524 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" podUID="54fff827-a0b0-4999-9f01-7d438d3270f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Jul 31 19:38:50.693565 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:50.693534 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:50.694048 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:50.693920 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:38:50.694638 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:38:50.694618 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:39:01.367921 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.367880 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh"] Jul 31 19:39:01.371801 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.371779 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:01.373670 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.373642 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-90f898-openshift-default-dockercfg-tv4vg\"" Jul 31 19:39:01.382114 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.382090 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh"] Jul 31 19:39:01.469751 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.469712 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:39:01.488753 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.488709 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-credential-socket\") pod \"e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:01.488941 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.488763 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-istio-envoy\") pod \"e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:01.488941 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.488822 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-istio-podinfo\") pod \"e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:01.488941 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.488885 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-istiod-ca-cert\") pod \"e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:01.488941 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.488920 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-istio-data\") pod \"e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:01.488941 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.488936 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-istio-token\") pod \"e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:01.489142 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.488975 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-workload-socket\") pod \"e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:01.489142 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.489022 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvwwj\" (UniqueName: \"kubernetes.io/projected/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-kube-api-access-mvwwj\") pod \"e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:01.489142 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.489065 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-workload-certs\") pod \"e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:01.565670 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.565628 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:39:01.590014 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.589977 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-istiod-ca-cert\") pod \"e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:01.590204 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.590027 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-istio-data\") pod \"e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:01.590204 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.590052 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-istio-token\") pod \"e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:01.590204 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.590070 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-workload-socket\") pod \"e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:01.590204 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.590114 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvwwj\" (UniqueName: \"kubernetes.io/projected/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-kube-api-access-mvwwj\") pod \"e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:01.590204 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.590162 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-workload-certs\") pod \"e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:01.590509 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.590323 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-credential-socket\") pod \"e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:01.590509 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.590374 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-istio-envoy\") pod \"e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:01.590509 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.590405 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-istio-podinfo\") pod \"e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:01.590672 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.590534 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-istio-data\") pod \"e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:01.590672 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.590553 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-workload-socket\") pod \"e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:01.590672 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.590607 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-workload-certs\") pod \"e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:01.590829 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.590757 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-credential-socket\") pod \"e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:01.591150 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.591120 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-istiod-ca-cert\") pod \"e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:01.593010 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.592990 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-istio-podinfo\") pod \"e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:01.593145 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.593048 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-istio-envoy\") pod \"e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:01.598892 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.598867 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-istio-token\") pod \"e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:01.599059 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.599034 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvwwj\" (UniqueName: \"kubernetes.io/projected/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-kube-api-access-mvwwj\") pod \"e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:01.682749 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.682672 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:01.684839 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.684813 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:39:01.828351 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.828317 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh"] Jul 31 19:39:01.831898 ip-10-0-139-39 kubenswrapper[2565]: W0731 19:39:01.831868 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode21f9ca4_b9a2_4470_b6c6_58b81cf76a2a.slice/crio-93844e0b8e4bbd213d8f4c3bd64f99e60d8d1ae82cb3c43310f94d3008f10eb7 WatchSource:0}: Error finding container 93844e0b8e4bbd213d8f4c3bd64f99e60d8d1ae82cb3c43310f94d3008f10eb7: Status 404 returned error can't find the container with id 93844e0b8e4bbd213d8f4c3bd64f99e60d8d1ae82cb3c43310f94d3008f10eb7 Jul 31 19:39:01.834335 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.834304 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:39:01.834491 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.834384 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:39:01.834491 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:01.834435 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:39:02.608697 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:02.608662 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" event={"ID":"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a","Type":"ContainerStarted","Data":"72fa3c375ebde3b988fb75fae0b61415fc6a962989d575a3ec9536be475234b3"} Jul 31 19:39:02.609084 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:02.608736 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" event={"ID":"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a","Type":"ContainerStarted","Data":"93844e0b8e4bbd213d8f4c3bd64f99e60d8d1ae82cb3c43310f94d3008f10eb7"} Jul 31 19:39:02.629714 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:02.629666 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" podStartSLOduration=1.6296528719999999 podStartE2EDuration="1.629652872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:39:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:39:02.628351281 +0000 UTC m=+3319.266341082" watchObservedRunningTime="2026-07-31 19:39:02.629652872 +0000 UTC m=+3319.267642653" Jul 31 19:39:02.683923 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:02.683892 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:02.689137 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:02.689110 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:03.614155 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:03.614128 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:03.615188 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:03.615170 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:04.520514 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:04.520483 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:39:06.417443 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:06.417367 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:39:09.465588 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:09.465551 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:39:18.882162 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:18.881496 2565 patch_prober.go:28] interesting pod/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": read tcp 10.133.0.2:54344->10.133.0.72:15021: read: connection reset by peer" start-of-body= Jul 31 19:39:18.882162 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:18.881563 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" podUID="e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": read tcp 10.133.0.2:54344->10.133.0.72:15021: read: connection reset by peer" Jul 31 19:39:19.705404 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:19.705371 2565 generic.go:356] "Generic (PLEG): container finished" podID="e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a" containerID="72fa3c375ebde3b988fb75fae0b61415fc6a962989d575a3ec9536be475234b3" exitCode=0 Jul 31 19:39:19.705575 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:19.705443 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" event={"ID":"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a","Type":"ContainerDied","Data":"72fa3c375ebde3b988fb75fae0b61415fc6a962989d575a3ec9536be475234b3"} Jul 31 19:39:19.705867 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:19.705852 2565 scope.go:117] "RemoveContainer" containerID="72fa3c375ebde3b988fb75fae0b61415fc6a962989d575a3ec9536be475234b3" Jul 31 19:39:19.706915 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:19.706886 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:39:19.707019 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:19.706973 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:39:19.707085 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:19.707023 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:39:20.712305 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:20.712269 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" event={"ID":"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a","Type":"ContainerStarted","Data":"8b9d4f2b2fc569280731986fb19cd694d9df1c12ea2082c7b0216f8bb4b851e8"} Jul 31 19:39:21.683349 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:21.683309 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:21.683619 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:21.683594 2565 patch_prober.go:28] interesting pod/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Jul 31 19:39:21.683714 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:21.683655 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" podUID="e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Jul 31 19:39:22.683185 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:22.683139 2565 patch_prober.go:28] interesting pod/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Jul 31 19:39:22.683593 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:22.683205 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" podUID="e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Jul 31 19:39:23.683680 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:23.683644 2565 patch_prober.go:28] interesting pod/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Jul 31 19:39:23.684067 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:23.683699 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" podUID="e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Jul 31 19:39:24.684018 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:24.683981 2565 patch_prober.go:28] interesting pod/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Jul 31 19:39:24.684433 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:24.684037 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" podUID="e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Jul 31 19:39:25.683298 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:25.683257 2565 patch_prober.go:28] interesting pod/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Jul 31 19:39:25.683482 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:25.683336 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" podUID="e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Jul 31 19:39:26.686675 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:26.686647 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:26.687115 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:26.686977 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:26.687746 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:26.687726 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:39:29.447688 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:29.447647 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:39:41.409162 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:41.409126 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:39:44.163027 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:39:44.162986 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:40:04.850465 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:04.850430 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:40:16.363997 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.363959 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:40:16.602365 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.602273 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m"] Jul 31 19:40:16.606055 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.606035 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:16.607964 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.607944 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-85e026-openshift-default-dockercfg-xs8bb\"" Jul 31 19:40:16.616131 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.616077 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m"] Jul 31 19:40:16.631809 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.631783 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:40:16.665472 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.665442 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-istio-podinfo\") pod \"e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:16.665577 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.665477 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-istiod-ca-cert\") pod \"e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:16.665577 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.665540 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-credential-socket\") pod \"e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:16.665661 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.665620 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n889d\" (UniqueName: \"kubernetes.io/projected/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-kube-api-access-n889d\") pod \"e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:16.665661 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.665646 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-istio-envoy\") pod \"e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:16.665752 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.665733 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-istio-token\") pod \"e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:16.665791 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.665771 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-workload-socket\") pod \"e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:16.665827 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.665800 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-workload-certs\") pod \"e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:16.665827 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.665819 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-istio-data\") pod \"e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:16.766720 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.766685 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-istio-token\") pod \"e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:16.766720 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.766728 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-workload-socket\") pod \"e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:16.767002 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.766758 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-workload-certs\") pod \"e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:16.767002 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.766777 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-istio-data\") pod \"e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:16.767002 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.766792 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-istio-podinfo\") pod \"e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:16.767002 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.766812 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-istiod-ca-cert\") pod \"e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:16.767002 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.766857 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-credential-socket\") pod \"e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:16.767002 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.766925 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n889d\" (UniqueName: \"kubernetes.io/projected/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-kube-api-access-n889d\") pod \"e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:16.767002 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.766952 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-istio-envoy\") pod \"e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:16.767423 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.767201 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-workload-certs\") pod \"e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:16.767423 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.767225 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-istio-data\") pod \"e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:16.767640 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.767602 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-workload-socket\") pod \"e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:16.767790 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.767770 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-credential-socket\") pod \"e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:16.768212 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.768187 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-istiod-ca-cert\") pod \"e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:16.769203 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.769185 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-istio-envoy\") pod \"e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:16.769588 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.769566 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-istio-podinfo\") pod \"e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:16.775943 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.775900 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-istio-token\") pod \"e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:16.776048 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.775966 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n889d\" (UniqueName: \"kubernetes.io/projected/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-kube-api-access-n889d\") pod \"e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:16.892487 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.892391 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:40:16.918348 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:16.918321 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:17.069683 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:17.069599 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m"] Jul 31 19:40:17.077621 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:17.077586 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:40:17.077732 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:17.077670 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:40:17.077732 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:17.077713 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:40:18.063713 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:18.063664 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" event={"ID":"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6","Type":"ContainerStarted","Data":"c7148ad7721561ed0512185276601984053e82267b18e4d760bc787317828a4b"} Jul 31 19:40:18.064126 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:18.063726 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" event={"ID":"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6","Type":"ContainerStarted","Data":"644853ebe32996834d3de18abcc33afc72c175d52bf83a2de988db18d8303c66"} Jul 31 19:40:18.098823 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:18.098772 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" podStartSLOduration=2.098756104 podStartE2EDuration="2.098756104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:40:18.098470455 +0000 UTC m=+3394.736460292" watchObservedRunningTime="2026-07-31 19:40:18.098756104 +0000 UTC m=+3394.736745959" Jul 31 19:40:18.918774 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:18.918733 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:18.923871 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:18.923841 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:19.069306 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:19.069278 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:19.070394 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:19.070367 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:20.095829 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:20.095795 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:40:20.197145 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:20.197113 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:40:21.141803 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:21.141770 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:40:21.436904 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:21.436819 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:40:21.612283 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:21.612208 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:40:21.709812 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:21.709729 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:40:26.652876 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:26.652838 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:40:33.460364 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:33.460332 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:40:35.110425 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:35.110378 2565 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 31 19:40:35.110939 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:35.110455 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" podUID="fbe41a06-fe16-4d89-b1f1-d98c6baab2a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 31 19:40:37.186372 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:37.186279 2565 generic.go:356] "Generic (PLEG): container finished" podID="fbe41a06-fe16-4d89-b1f1-d98c6baab2a6" containerID="c7148ad7721561ed0512185276601984053e82267b18e4d760bc787317828a4b" exitCode=0 Jul 31 19:40:37.186372 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:37.186331 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" event={"ID":"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6","Type":"ContainerDied","Data":"c7148ad7721561ed0512185276601984053e82267b18e4d760bc787317828a4b"} Jul 31 19:40:37.186940 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:37.186823 2565 scope.go:117] "RemoveContainer" containerID="c7148ad7721561ed0512185276601984053e82267b18e4d760bc787317828a4b" Jul 31 19:40:37.187954 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:37.187924 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:40:37.188126 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:37.188006 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:40:37.188126 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:37.188055 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 31 19:40:38.193919 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:38.193885 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" event={"ID":"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6","Type":"ContainerStarted","Data":"e0515a0b2bca21eb95c00d2714f91e92d080d0192462c6a462b88b0da6baa471"} Jul 31 19:40:38.919010 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:38.918967 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:38.919234 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:38.919210 2565 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 31 19:40:38.919329 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:38.919284 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" podUID="fbe41a06-fe16-4d89-b1f1-d98c6baab2a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 31 19:40:39.919062 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:39.919026 2565 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 31 19:40:39.919459 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:39.919077 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" podUID="fbe41a06-fe16-4d89-b1f1-d98c6baab2a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 31 19:40:40.919488 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:40.919456 2565 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 31 19:40:40.919855 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:40.919519 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" podUID="fbe41a06-fe16-4d89-b1f1-d98c6baab2a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 31 19:40:41.918993 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:41.918957 2565 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 31 19:40:41.919193 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:41.919014 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" podUID="fbe41a06-fe16-4d89-b1f1-d98c6baab2a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 31 19:40:42.919167 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:42.919130 2565 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 31 19:40:42.919603 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:42.919198 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" podUID="fbe41a06-fe16-4d89-b1f1-d98c6baab2a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 31 19:40:43.922282 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:43.922232 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:43.922653 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:43.922598 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:43.923279 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:43.923259 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:40:44.920473 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:44.920438 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:40:51.804888 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:51.804853 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:40:51.904949 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:51.904918 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:40:51.992813 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:51.992775 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:40:52.062985 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:52.062904 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:40:52.365006 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:52.364910 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:40:54.955032 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:54.954997 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:40:58.335266 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:58.335218 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:40:58.402249 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:58.402202 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:40:58.574880 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:58.574843 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m"] Jul 31 19:40:58.575139 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:58.575112 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" podUID="fbe41a06-fe16-4d89-b1f1-d98c6baab2a6" containerName="istio-proxy" containerID="cri-o://e0515a0b2bca21eb95c00d2714f91e92d080d0192462c6a462b88b0da6baa471" gracePeriod=30 Jul 31 19:40:58.588133 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:58.588069 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:40:58.801423 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:58.801380 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:40:58.924936 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:58.924847 2565 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 31 19:40:58.924936 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:40:58.924913 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" podUID="fbe41a06-fe16-4d89-b1f1-d98c6baab2a6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 31 19:41:03.838178 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:03.838149 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:41:03.958061 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:03.957973 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-istio-podinfo\") pod \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " Jul 31 19:41:03.958061 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:03.958026 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-istio-envoy\") pod \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " Jul 31 19:41:03.958306 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:03.958059 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-credential-socket\") pod \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " Jul 31 19:41:03.958306 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:03.958095 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n889d\" (UniqueName: \"kubernetes.io/projected/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-kube-api-access-n889d\") pod \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " Jul 31 19:41:03.958306 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:03.958151 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-istiod-ca-cert\") pod \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " Jul 31 19:41:03.958306 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:03.958196 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-workload-socket\") pod \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " Jul 31 19:41:03.958306 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:03.958226 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-istio-token\") pod \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " Jul 31 19:41:03.958306 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:03.958300 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-istio-data\") pod \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " Jul 31 19:41:03.959022 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:03.958327 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-workload-certs\") pod \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\" (UID: \"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6\") " Jul 31 19:41:03.959022 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:03.958438 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fbe41a06-fe16-4d89-b1f1-d98c6baab2a6" (UID: "fbe41a06-fe16-4d89-b1f1-d98c6baab2a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:41:03.959022 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:03.958686 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fbe41a06-fe16-4d89-b1f1-d98c6baab2a6" (UID: "fbe41a06-fe16-4d89-b1f1-d98c6baab2a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:41:03.959022 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:03.958701 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fbe41a06-fe16-4d89-b1f1-d98c6baab2a6" (UID: "fbe41a06-fe16-4d89-b1f1-d98c6baab2a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 19:41:03.959022 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:03.958885 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-istio-data" (OuterVolumeSpecName: "istio-data") pod "fbe41a06-fe16-4d89-b1f1-d98c6baab2a6" (UID: "fbe41a06-fe16-4d89-b1f1-d98c6baab2a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:41:03.959022 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:03.958895 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fbe41a06-fe16-4d89-b1f1-d98c6baab2a6" (UID: "fbe41a06-fe16-4d89-b1f1-d98c6baab2a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:41:03.959022 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:03.959016 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-credential-socket\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:41:03.959344 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:03.959036 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-istiod-ca-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:41:03.959344 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:03.959051 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-workload-socket\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:41:03.959344 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:03.959064 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-istio-data\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:41:03.959344 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:03.959077 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-workload-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:41:03.960486 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:03.960437 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fbe41a06-fe16-4d89-b1f1-d98c6baab2a6" (UID: "fbe41a06-fe16-4d89-b1f1-d98c6baab2a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:41:03.960603 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:03.960560 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-kube-api-access-n889d" (OuterVolumeSpecName: "kube-api-access-n889d") pod "fbe41a06-fe16-4d89-b1f1-d98c6baab2a6" (UID: "fbe41a06-fe16-4d89-b1f1-d98c6baab2a6"). InnerVolumeSpecName "kube-api-access-n889d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:41:03.960918 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:03.960896 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fbe41a06-fe16-4d89-b1f1-d98c6baab2a6" (UID: "fbe41a06-fe16-4d89-b1f1-d98c6baab2a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 19:41:03.961062 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:03.961041 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-istio-token" (OuterVolumeSpecName: "istio-token") pod "fbe41a06-fe16-4d89-b1f1-d98c6baab2a6" (UID: "fbe41a06-fe16-4d89-b1f1-d98c6baab2a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:41:04.060474 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:04.060440 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-istio-token\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:41:04.060474 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:04.060467 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-istio-podinfo\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:41:04.060474 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:04.060477 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-istio-envoy\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:41:04.060735 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:04.060486 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n889d\" (UniqueName: \"kubernetes.io/projected/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6-kube-api-access-n889d\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:41:04.334286 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:04.334224 2565 generic.go:356] "Generic (PLEG): container finished" podID="fbe41a06-fe16-4d89-b1f1-d98c6baab2a6" containerID="e0515a0b2bca21eb95c00d2714f91e92d080d0192462c6a462b88b0da6baa471" exitCode=0 Jul 31 19:41:04.334286 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:04.334287 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" Jul 31 19:41:04.334498 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:04.334312 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" event={"ID":"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6","Type":"ContainerDied","Data":"e0515a0b2bca21eb95c00d2714f91e92d080d0192462c6a462b88b0da6baa471"} Jul 31 19:41:04.334498 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:04.334370 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m" event={"ID":"fbe41a06-fe16-4d89-b1f1-d98c6baab2a6","Type":"ContainerDied","Data":"644853ebe32996834d3de18abcc33afc72c175d52bf83a2de988db18d8303c66"} Jul 31 19:41:04.334498 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:04.334392 2565 scope.go:117] "RemoveContainer" containerID="e0515a0b2bca21eb95c00d2714f91e92d080d0192462c6a462b88b0da6baa471" Jul 31 19:41:04.344826 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:04.344806 2565 scope.go:117] "RemoveContainer" containerID="c7148ad7721561ed0512185276601984053e82267b18e4d760bc787317828a4b" Jul 31 19:41:04.356376 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:04.356293 2565 scope.go:117] "RemoveContainer" containerID="e0515a0b2bca21eb95c00d2714f91e92d080d0192462c6a462b88b0da6baa471" Jul 31 19:41:04.357003 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:41:04.356720 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0515a0b2bca21eb95c00d2714f91e92d080d0192462c6a462b88b0da6baa471\": container with ID starting with e0515a0b2bca21eb95c00d2714f91e92d080d0192462c6a462b88b0da6baa471 not found: ID does not exist" containerID="e0515a0b2bca21eb95c00d2714f91e92d080d0192462c6a462b88b0da6baa471" Jul 31 19:41:04.357003 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:04.356778 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0515a0b2bca21eb95c00d2714f91e92d080d0192462c6a462b88b0da6baa471"} err="failed to get container status \"e0515a0b2bca21eb95c00d2714f91e92d080d0192462c6a462b88b0da6baa471\": rpc error: code = NotFound desc = could not find container \"e0515a0b2bca21eb95c00d2714f91e92d080d0192462c6a462b88b0da6baa471\": container with ID starting with e0515a0b2bca21eb95c00d2714f91e92d080d0192462c6a462b88b0da6baa471 not found: ID does not exist" Jul 31 19:41:04.357003 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:04.356805 2565 scope.go:117] "RemoveContainer" containerID="c7148ad7721561ed0512185276601984053e82267b18e4d760bc787317828a4b" Jul 31 19:41:04.357188 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:41:04.357145 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7148ad7721561ed0512185276601984053e82267b18e4d760bc787317828a4b\": container with ID starting with c7148ad7721561ed0512185276601984053e82267b18e4d760bc787317828a4b not found: ID does not exist" containerID="c7148ad7721561ed0512185276601984053e82267b18e4d760bc787317828a4b" Jul 31 19:41:04.357188 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:04.357169 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7148ad7721561ed0512185276601984053e82267b18e4d760bc787317828a4b"} err="failed to get container status \"c7148ad7721561ed0512185276601984053e82267b18e4d760bc787317828a4b\": rpc error: code = NotFound desc = could not find container \"c7148ad7721561ed0512185276601984053e82267b18e4d760bc787317828a4b\": container with ID starting with c7148ad7721561ed0512185276601984053e82267b18e4d760bc787317828a4b not found: ID does not exist" Jul 31 19:41:04.359479 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:04.359451 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m"] Jul 31 19:41:04.364915 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:04.364894 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-85e026-openshift-default-df86c5b4d-dhs7m"] Jul 31 19:41:05.887459 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:05.887426 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbe41a06-fe16-4d89-b1f1-d98c6baab2a6" path="/var/lib/kubelet/pods/fbe41a06-fe16-4d89-b1f1-d98c6baab2a6/volumes" Jul 31 19:41:09.648554 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:09.648517 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:41:11.821443 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:11.821407 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:41:11.892004 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:11.891971 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:41:11.953315 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:11.953280 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:41:12.054183 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:12.054142 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:41:12.346567 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:12.346534 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:41:16.963940 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:16.963856 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:41:17.444581 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:17.444532 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:41:46.834481 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:46.834441 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:41:46.891699 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:46.891648 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:41:54.332963 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:54.332916 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:41:54.649269 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:54.649156 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:41:58.612805 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:41:58.612772 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:42:00.541064 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:00.541024 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:42:06.470734 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:06.470652 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:42:06.527782 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:06.527749 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:42:06.714478 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:06.714442 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk"] Jul 31 19:42:06.714730 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:06.714704 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" podUID="54fff827-a0b0-4999-9f01-7d438d3270f4" containerName="istio-proxy" containerID="cri-o://da619dfa18a07cd0f904bb2cb3d92b5c11c527ddfcea49b6764165f373812bf8" gracePeriod=30 Jul 31 19:42:06.732937 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:06.732871 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:42:06.969046 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:06.969009 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:42:11.971583 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:11.971555 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:42:12.052186 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.052154 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54fff827-a0b0-4999-9f01-7d438d3270f4-istiod-ca-cert\") pod \"54fff827-a0b0-4999-9f01-7d438d3270f4\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " Jul 31 19:42:12.052366 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.052217 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54fff827-a0b0-4999-9f01-7d438d3270f4-istio-token\") pod \"54fff827-a0b0-4999-9f01-7d438d3270f4\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " Jul 31 19:42:12.052366 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.052297 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vksq\" (UniqueName: \"kubernetes.io/projected/54fff827-a0b0-4999-9f01-7d438d3270f4-kube-api-access-4vksq\") pod \"54fff827-a0b0-4999-9f01-7d438d3270f4\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " Jul 31 19:42:12.052366 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.052313 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54fff827-a0b0-4999-9f01-7d438d3270f4-istio-envoy\") pod \"54fff827-a0b0-4999-9f01-7d438d3270f4\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " Jul 31 19:42:12.052366 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.052349 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fff827-a0b0-4999-9f01-7d438d3270f4-credential-socket\") pod \"54fff827-a0b0-4999-9f01-7d438d3270f4\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " Jul 31 19:42:12.052616 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.052375 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54fff827-a0b0-4999-9f01-7d438d3270f4-istio-data\") pod \"54fff827-a0b0-4999-9f01-7d438d3270f4\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " Jul 31 19:42:12.052616 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.052429 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54fff827-a0b0-4999-9f01-7d438d3270f4-istio-podinfo\") pod \"54fff827-a0b0-4999-9f01-7d438d3270f4\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " Jul 31 19:42:12.052616 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.052446 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fff827-a0b0-4999-9f01-7d438d3270f4-workload-socket\") pod \"54fff827-a0b0-4999-9f01-7d438d3270f4\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " Jul 31 19:42:12.052616 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.052468 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54fff827-a0b0-4999-9f01-7d438d3270f4-workload-certs\") pod \"54fff827-a0b0-4999-9f01-7d438d3270f4\" (UID: \"54fff827-a0b0-4999-9f01-7d438d3270f4\") " Jul 31 19:42:12.052889 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.052707 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fff827-a0b0-4999-9f01-7d438d3270f4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "54fff827-a0b0-4999-9f01-7d438d3270f4" (UID: "54fff827-a0b0-4999-9f01-7d438d3270f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:42:12.052889 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.052733 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54fff827-a0b0-4999-9f01-7d438d3270f4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "54fff827-a0b0-4999-9f01-7d438d3270f4" (UID: "54fff827-a0b0-4999-9f01-7d438d3270f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 19:42:12.052996 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.052888 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fff827-a0b0-4999-9f01-7d438d3270f4-credential-socket\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:42:12.052996 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.052907 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54fff827-a0b0-4999-9f01-7d438d3270f4-istiod-ca-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:42:12.052996 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.052935 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fff827-a0b0-4999-9f01-7d438d3270f4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "54fff827-a0b0-4999-9f01-7d438d3270f4" (UID: "54fff827-a0b0-4999-9f01-7d438d3270f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:42:12.052996 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.052950 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fff827-a0b0-4999-9f01-7d438d3270f4-istio-data" (OuterVolumeSpecName: "istio-data") pod "54fff827-a0b0-4999-9f01-7d438d3270f4" (UID: "54fff827-a0b0-4999-9f01-7d438d3270f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:42:12.052996 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.052959 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fff827-a0b0-4999-9f01-7d438d3270f4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "54fff827-a0b0-4999-9f01-7d438d3270f4" (UID: "54fff827-a0b0-4999-9f01-7d438d3270f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:42:12.054682 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.054655 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fff827-a0b0-4999-9f01-7d438d3270f4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "54fff827-a0b0-4999-9f01-7d438d3270f4" (UID: "54fff827-a0b0-4999-9f01-7d438d3270f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:42:12.054811 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.054752 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54fff827-a0b0-4999-9f01-7d438d3270f4-kube-api-access-4vksq" (OuterVolumeSpecName: "kube-api-access-4vksq") pod "54fff827-a0b0-4999-9f01-7d438d3270f4" (UID: "54fff827-a0b0-4999-9f01-7d438d3270f4"). InnerVolumeSpecName "kube-api-access-4vksq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:42:12.054981 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.054965 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54fff827-a0b0-4999-9f01-7d438d3270f4-istio-token" (OuterVolumeSpecName: "istio-token") pod "54fff827-a0b0-4999-9f01-7d438d3270f4" (UID: "54fff827-a0b0-4999-9f01-7d438d3270f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:42:12.055029 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.054985 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54fff827-a0b0-4999-9f01-7d438d3270f4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "54fff827-a0b0-4999-9f01-7d438d3270f4" (UID: "54fff827-a0b0-4999-9f01-7d438d3270f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 19:42:12.153697 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.153668 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54fff827-a0b0-4999-9f01-7d438d3270f4-istio-token\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:42:12.153697 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.153695 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vksq\" (UniqueName: \"kubernetes.io/projected/54fff827-a0b0-4999-9f01-7d438d3270f4-kube-api-access-4vksq\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:42:12.153893 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.153707 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54fff827-a0b0-4999-9f01-7d438d3270f4-istio-envoy\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:42:12.153893 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.153719 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54fff827-a0b0-4999-9f01-7d438d3270f4-istio-data\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:42:12.153893 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.153727 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54fff827-a0b0-4999-9f01-7d438d3270f4-istio-podinfo\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:42:12.153893 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.153734 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fff827-a0b0-4999-9f01-7d438d3270f4-workload-socket\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:42:12.153893 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.153744 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54fff827-a0b0-4999-9f01-7d438d3270f4-workload-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:42:12.696743 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.696706 2565 generic.go:356] "Generic (PLEG): container finished" podID="54fff827-a0b0-4999-9f01-7d438d3270f4" containerID="da619dfa18a07cd0f904bb2cb3d92b5c11c527ddfcea49b6764165f373812bf8" exitCode=0 Jul 31 19:42:12.696949 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.696763 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" Jul 31 19:42:12.696949 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.696783 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" event={"ID":"54fff827-a0b0-4999-9f01-7d438d3270f4","Type":"ContainerDied","Data":"da619dfa18a07cd0f904bb2cb3d92b5c11c527ddfcea49b6764165f373812bf8"} Jul 31 19:42:12.696949 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.696856 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk" event={"ID":"54fff827-a0b0-4999-9f01-7d438d3270f4","Type":"ContainerDied","Data":"ccb4dce655a71e3a05afdf2f8accfd25f65a39a3c3806beec4b187ffb3e19b69"} Jul 31 19:42:12.696949 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.696878 2565 scope.go:117] "RemoveContainer" containerID="da619dfa18a07cd0f904bb2cb3d92b5c11c527ddfcea49b6764165f373812bf8" Jul 31 19:42:12.706475 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.706458 2565 scope.go:117] "RemoveContainer" containerID="6045fa43c551954a9d7afb0596d1d3cac211f9139e4b2f18832370dfcdc53447" Jul 31 19:42:12.714638 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.714619 2565 scope.go:117] "RemoveContainer" containerID="da619dfa18a07cd0f904bb2cb3d92b5c11c527ddfcea49b6764165f373812bf8" Jul 31 19:42:12.714915 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:42:12.714896 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da619dfa18a07cd0f904bb2cb3d92b5c11c527ddfcea49b6764165f373812bf8\": container with ID starting with da619dfa18a07cd0f904bb2cb3d92b5c11c527ddfcea49b6764165f373812bf8 not found: ID does not exist" containerID="da619dfa18a07cd0f904bb2cb3d92b5c11c527ddfcea49b6764165f373812bf8" Jul 31 19:42:12.714959 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.714924 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da619dfa18a07cd0f904bb2cb3d92b5c11c527ddfcea49b6764165f373812bf8"} err="failed to get container status \"da619dfa18a07cd0f904bb2cb3d92b5c11c527ddfcea49b6764165f373812bf8\": rpc error: code = NotFound desc = could not find container \"da619dfa18a07cd0f904bb2cb3d92b5c11c527ddfcea49b6764165f373812bf8\": container with ID starting with da619dfa18a07cd0f904bb2cb3d92b5c11c527ddfcea49b6764165f373812bf8 not found: ID does not exist" Jul 31 19:42:12.714959 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.714942 2565 scope.go:117] "RemoveContainer" containerID="6045fa43c551954a9d7afb0596d1d3cac211f9139e4b2f18832370dfcdc53447" Jul 31 19:42:12.715186 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:42:12.715169 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6045fa43c551954a9d7afb0596d1d3cac211f9139e4b2f18832370dfcdc53447\": container with ID starting with 6045fa43c551954a9d7afb0596d1d3cac211f9139e4b2f18832370dfcdc53447 not found: ID does not exist" containerID="6045fa43c551954a9d7afb0596d1d3cac211f9139e4b2f18832370dfcdc53447" Jul 31 19:42:12.715225 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.715192 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6045fa43c551954a9d7afb0596d1d3cac211f9139e4b2f18832370dfcdc53447"} err="failed to get container status \"6045fa43c551954a9d7afb0596d1d3cac211f9139e4b2f18832370dfcdc53447\": rpc error: code = NotFound desc = could not find container \"6045fa43c551954a9d7afb0596d1d3cac211f9139e4b2f18832370dfcdc53447\": container with ID starting with 6045fa43c551954a9d7afb0596d1d3cac211f9139e4b2f18832370dfcdc53447 not found: ID does not exist" Jul 31 19:42:12.719000 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.718978 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk"] Jul 31 19:42:12.722894 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:12.722873 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-0368ff-openshift-default-77b9f9889b-5dhpk"] Jul 31 19:42:13.886532 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:13.886499 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54fff827-a0b0-4999-9f01-7d438d3270f4" path="/var/lib/kubelet/pods/54fff827-a0b0-4999-9f01-7d438d3270f4/volumes" Jul 31 19:42:17.565073 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:17.565039 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:42:17.666382 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:17.666347 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:42:17.803557 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:17.803509 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:42:17.864498 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:17.864395 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:42:18.168986 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:18.168878 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:42:20.162905 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:20.162866 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:42:24.213041 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:24.213008 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:42:24.274643 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:24.274615 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:42:24.462520 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:24.462487 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh"] Jul 31 19:42:24.462746 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:24.462722 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" podUID="e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a" containerName="istio-proxy" containerID="cri-o://8b9d4f2b2fc569280731986fb19cd694d9df1c12ea2082c7b0216f8bb4b851e8" gracePeriod=30 Jul 31 19:42:24.491411 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:24.491380 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:42:24.693057 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:24.693022 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:42:25.361199 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:25.361168 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:42:25.446947 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:25.446916 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:42:26.689771 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:26.689736 2565 patch_prober.go:28] interesting pod/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 31 19:42:26.690160 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:26.689789 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" podUID="e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 31 19:42:29.719334 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.719302 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:42:29.793121 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.793090 2565 generic.go:356] "Generic (PLEG): container finished" podID="e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a" containerID="8b9d4f2b2fc569280731986fb19cd694d9df1c12ea2082c7b0216f8bb4b851e8" exitCode=0 Jul 31 19:42:29.793325 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.793131 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" event={"ID":"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a","Type":"ContainerDied","Data":"8b9d4f2b2fc569280731986fb19cd694d9df1c12ea2082c7b0216f8bb4b851e8"} Jul 31 19:42:29.793325 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.793146 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" Jul 31 19:42:29.793325 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.793172 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh" event={"ID":"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a","Type":"ContainerDied","Data":"93844e0b8e4bbd213d8f4c3bd64f99e60d8d1ae82cb3c43310f94d3008f10eb7"} Jul 31 19:42:29.793325 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.793195 2565 scope.go:117] "RemoveContainer" containerID="8b9d4f2b2fc569280731986fb19cd694d9df1c12ea2082c7b0216f8bb4b851e8" Jul 31 19:42:29.801919 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.801899 2565 scope.go:117] "RemoveContainer" containerID="72fa3c375ebde3b988fb75fae0b61415fc6a962989d575a3ec9536be475234b3" Jul 31 19:42:29.810203 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.810184 2565 scope.go:117] "RemoveContainer" containerID="8b9d4f2b2fc569280731986fb19cd694d9df1c12ea2082c7b0216f8bb4b851e8" Jul 31 19:42:29.810469 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:42:29.810450 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b9d4f2b2fc569280731986fb19cd694d9df1c12ea2082c7b0216f8bb4b851e8\": container with ID starting with 8b9d4f2b2fc569280731986fb19cd694d9df1c12ea2082c7b0216f8bb4b851e8 not found: ID does not exist" containerID="8b9d4f2b2fc569280731986fb19cd694d9df1c12ea2082c7b0216f8bb4b851e8" Jul 31 19:42:29.810535 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.810478 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b9d4f2b2fc569280731986fb19cd694d9df1c12ea2082c7b0216f8bb4b851e8"} err="failed to get container status \"8b9d4f2b2fc569280731986fb19cd694d9df1c12ea2082c7b0216f8bb4b851e8\": rpc error: code = NotFound desc = could not find container \"8b9d4f2b2fc569280731986fb19cd694d9df1c12ea2082c7b0216f8bb4b851e8\": container with ID starting with 8b9d4f2b2fc569280731986fb19cd694d9df1c12ea2082c7b0216f8bb4b851e8 not found: ID does not exist" Jul 31 19:42:29.810535 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.810494 2565 scope.go:117] "RemoveContainer" containerID="72fa3c375ebde3b988fb75fae0b61415fc6a962989d575a3ec9536be475234b3" Jul 31 19:42:29.810751 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:42:29.810731 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72fa3c375ebde3b988fb75fae0b61415fc6a962989d575a3ec9536be475234b3\": container with ID starting with 72fa3c375ebde3b988fb75fae0b61415fc6a962989d575a3ec9536be475234b3 not found: ID does not exist" containerID="72fa3c375ebde3b988fb75fae0b61415fc6a962989d575a3ec9536be475234b3" Jul 31 19:42:29.810811 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.810760 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72fa3c375ebde3b988fb75fae0b61415fc6a962989d575a3ec9536be475234b3"} err="failed to get container status \"72fa3c375ebde3b988fb75fae0b61415fc6a962989d575a3ec9536be475234b3\": rpc error: code = NotFound desc = could not find container \"72fa3c375ebde3b988fb75fae0b61415fc6a962989d575a3ec9536be475234b3\": container with ID starting with 72fa3c375ebde3b988fb75fae0b61415fc6a962989d575a3ec9536be475234b3 not found: ID does not exist" Jul 31 19:42:29.828213 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.828191 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-istiod-ca-cert\") pod \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " Jul 31 19:42:29.828339 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.828225 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-istio-data\") pod \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " Jul 31 19:42:29.828339 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.828271 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-workload-socket\") pod \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " Jul 31 19:42:29.828339 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.828298 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvwwj\" (UniqueName: \"kubernetes.io/projected/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-kube-api-access-mvwwj\") pod \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " Jul 31 19:42:29.828494 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.828340 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-credential-socket\") pod \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " Jul 31 19:42:29.828494 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.828377 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-workload-certs\") pod \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " Jul 31 19:42:29.828494 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.828427 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-istio-podinfo\") pod \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " Jul 31 19:42:29.828494 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.828448 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-istio-envoy\") pod \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " Jul 31 19:42:29.828494 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.828463 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-istio-token\") pod \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\" (UID: \"e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a\") " Jul 31 19:42:29.828757 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.828592 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a" (UID: "e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:42:29.828757 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.828643 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a" (UID: "e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 19:42:29.828757 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.828647 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a" (UID: "e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:42:29.828927 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.828819 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-istiod-ca-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:42:29.828927 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.828827 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a" (UID: "e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:42:29.828927 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.828836 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-workload-socket\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:42:29.828927 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.828880 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-credential-socket\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:42:29.828927 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.828913 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-istio-data" (OuterVolumeSpecName: "istio-data") pod "e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a" (UID: "e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:42:29.830618 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.830592 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a" (UID: "e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 19:42:29.830693 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.830629 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-istio-token" (OuterVolumeSpecName: "istio-token") pod "e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a" (UID: "e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:42:29.830877 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.830852 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a" (UID: "e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:42:29.830966 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.830852 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-kube-api-access-mvwwj" (OuterVolumeSpecName: "kube-api-access-mvwwj") pod "e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a" (UID: "e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a"). InnerVolumeSpecName "kube-api-access-mvwwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:42:29.929581 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.929550 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-workload-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:42:29.929581 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.929576 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-istio-podinfo\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:42:29.929581 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.929584 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-istio-envoy\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:42:29.929801 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.929592 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-istio-token\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:42:29.929801 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.929599 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-istio-data\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:42:29.929801 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:29.929628 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvwwj\" (UniqueName: \"kubernetes.io/projected/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a-kube-api-access-mvwwj\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:42:30.112374 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:30.112298 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh"] Jul 31 19:42:30.115717 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:30.115691 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-90f898-openshift-default-7dbcfb479b-wklfh"] Jul 31 19:42:31.887499 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:31.887465 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a" path="/var/lib/kubelet/pods/e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a/volumes" Jul 31 19:42:35.420496 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:35.420465 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:42:35.500983 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:35.500956 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:42:35.576862 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:35.576833 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:42:35.677412 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:35.677334 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:42:36.063947 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:36.063915 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:42:37.759983 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:37.759937 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:42:42.363703 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:42.363670 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:42:42.427726 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:42.427695 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:42:42.590018 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:42.589980 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw"] Jul 31 19:42:42.590278 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:42.590218 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" podUID="3f0b2bc3-d509-461c-90a5-0d68eb70e012" containerName="istio-proxy" containerID="cri-o://7cc703fc8c932fd59f0647736d45a8d6cb558093975e32ae854e89beafe77b6e" gracePeriod=30 Jul 31 19:42:42.612985 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:42.612941 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:42:42.824123 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:42.824086 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:42:45.553234 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:45.553191 2565 patch_prober.go:28] interesting pod/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 31 19:42:45.553658 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:45.553279 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" podUID="3f0b2bc3-d509-461c-90a5-0d68eb70e012" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 31 19:42:47.847552 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.847521 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:42:47.891670 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.891585 2565 generic.go:356] "Generic (PLEG): container finished" podID="3f0b2bc3-d509-461c-90a5-0d68eb70e012" containerID="7cc703fc8c932fd59f0647736d45a8d6cb558093975e32ae854e89beafe77b6e" exitCode=0 Jul 31 19:42:47.891670 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.891643 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" Jul 31 19:42:47.891877 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.891669 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" event={"ID":"3f0b2bc3-d509-461c-90a5-0d68eb70e012","Type":"ContainerDied","Data":"7cc703fc8c932fd59f0647736d45a8d6cb558093975e32ae854e89beafe77b6e"} Jul 31 19:42:47.891877 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.891718 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw" event={"ID":"3f0b2bc3-d509-461c-90a5-0d68eb70e012","Type":"ContainerDied","Data":"ed23778600ed1e4e8ca20dfe82ffa968fc9dd78321d9d1b6c3e99f60a426507b"} Jul 31 19:42:47.891877 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.891741 2565 scope.go:117] "RemoveContainer" containerID="7cc703fc8c932fd59f0647736d45a8d6cb558093975e32ae854e89beafe77b6e" Jul 31 19:42:47.897507 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.897481 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f0b2bc3-d509-461c-90a5-0d68eb70e012-istio-envoy\") pod \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " Jul 31 19:42:47.897629 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.897515 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f0b2bc3-d509-461c-90a5-0d68eb70e012-istio-data\") pod \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " Jul 31 19:42:47.897629 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.897545 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0b2bc3-d509-461c-90a5-0d68eb70e012-workload-socket\") pod \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " Jul 31 19:42:47.897629 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.897592 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f0b2bc3-d509-461c-90a5-0d68eb70e012-workload-certs\") pod \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " Jul 31 19:42:47.897629 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.897619 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f0b2bc3-d509-461c-90a5-0d68eb70e012-istio-podinfo\") pod \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " Jul 31 19:42:47.897846 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.897661 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f0b2bc3-d509-461c-90a5-0d68eb70e012-istiod-ca-cert\") pod \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " Jul 31 19:42:47.897846 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.897706 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0b2bc3-d509-461c-90a5-0d68eb70e012-credential-socket\") pod \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " Jul 31 19:42:47.897846 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.897746 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzhv7\" (UniqueName: \"kubernetes.io/projected/3f0b2bc3-d509-461c-90a5-0d68eb70e012-kube-api-access-kzhv7\") pod \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " Jul 31 19:42:47.897846 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.897808 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f0b2bc3-d509-461c-90a5-0d68eb70e012-istio-data" (OuterVolumeSpecName: "istio-data") pod "3f0b2bc3-d509-461c-90a5-0d68eb70e012" (UID: "3f0b2bc3-d509-461c-90a5-0d68eb70e012"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:42:47.898057 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.897928 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f0b2bc3-d509-461c-90a5-0d68eb70e012-istio-token\") pod \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\" (UID: \"3f0b2bc3-d509-461c-90a5-0d68eb70e012\") " Jul 31 19:42:47.898138 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.898108 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f0b2bc3-d509-461c-90a5-0d68eb70e012-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3f0b2bc3-d509-461c-90a5-0d68eb70e012" (UID: "3f0b2bc3-d509-461c-90a5-0d68eb70e012"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:42:47.898206 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.898146 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f0b2bc3-d509-461c-90a5-0d68eb70e012-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3f0b2bc3-d509-461c-90a5-0d68eb70e012" (UID: "3f0b2bc3-d509-461c-90a5-0d68eb70e012"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 19:42:47.898206 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.898194 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f0b2bc3-d509-461c-90a5-0d68eb70e012-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3f0b2bc3-d509-461c-90a5-0d68eb70e012" (UID: "3f0b2bc3-d509-461c-90a5-0d68eb70e012"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:42:47.898429 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.898403 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f0b2bc3-d509-461c-90a5-0d68eb70e012-istiod-ca-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:42:47.898656 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.898436 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0b2bc3-d509-461c-90a5-0d68eb70e012-credential-socket\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:42:47.898656 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.898455 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f0b2bc3-d509-461c-90a5-0d68eb70e012-istio-data\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:42:47.898656 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.898507 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0b2bc3-d509-461c-90a5-0d68eb70e012-workload-socket\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:42:47.898656 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.898529 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f0b2bc3-d509-461c-90a5-0d68eb70e012-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3f0b2bc3-d509-461c-90a5-0d68eb70e012" (UID: "3f0b2bc3-d509-461c-90a5-0d68eb70e012"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:42:47.900186 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.900130 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f0b2bc3-d509-461c-90a5-0d68eb70e012-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3f0b2bc3-d509-461c-90a5-0d68eb70e012" (UID: "3f0b2bc3-d509-461c-90a5-0d68eb70e012"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:42:47.900321 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.900207 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f0b2bc3-d509-461c-90a5-0d68eb70e012-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3f0b2bc3-d509-461c-90a5-0d68eb70e012" (UID: "3f0b2bc3-d509-461c-90a5-0d68eb70e012"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 19:42:47.900553 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.900522 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f0b2bc3-d509-461c-90a5-0d68eb70e012-istio-token" (OuterVolumeSpecName: "istio-token") pod "3f0b2bc3-d509-461c-90a5-0d68eb70e012" (UID: "3f0b2bc3-d509-461c-90a5-0d68eb70e012"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:42:47.900657 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.900617 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f0b2bc3-d509-461c-90a5-0d68eb70e012-kube-api-access-kzhv7" (OuterVolumeSpecName: "kube-api-access-kzhv7") pod "3f0b2bc3-d509-461c-90a5-0d68eb70e012" (UID: "3f0b2bc3-d509-461c-90a5-0d68eb70e012"). InnerVolumeSpecName "kube-api-access-kzhv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:42:47.900957 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.900934 2565 scope.go:117] "RemoveContainer" containerID="c81137c96b254afdc7a6524989e0eabb780924be24e268e7e3d8270a5fd64571" Jul 31 19:42:47.919538 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.919518 2565 scope.go:117] "RemoveContainer" containerID="7cc703fc8c932fd59f0647736d45a8d6cb558093975e32ae854e89beafe77b6e" Jul 31 19:42:47.919822 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:42:47.919802 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cc703fc8c932fd59f0647736d45a8d6cb558093975e32ae854e89beafe77b6e\": container with ID starting with 7cc703fc8c932fd59f0647736d45a8d6cb558093975e32ae854e89beafe77b6e not found: ID does not exist" containerID="7cc703fc8c932fd59f0647736d45a8d6cb558093975e32ae854e89beafe77b6e" Jul 31 19:42:47.919875 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.919832 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cc703fc8c932fd59f0647736d45a8d6cb558093975e32ae854e89beafe77b6e"} err="failed to get container status \"7cc703fc8c932fd59f0647736d45a8d6cb558093975e32ae854e89beafe77b6e\": rpc error: code = NotFound desc = could not find container \"7cc703fc8c932fd59f0647736d45a8d6cb558093975e32ae854e89beafe77b6e\": container with ID starting with 7cc703fc8c932fd59f0647736d45a8d6cb558093975e32ae854e89beafe77b6e not found: ID does not exist" Jul 31 19:42:47.919875 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.919852 2565 scope.go:117] "RemoveContainer" containerID="c81137c96b254afdc7a6524989e0eabb780924be24e268e7e3d8270a5fd64571" Jul 31 19:42:47.920086 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:42:47.920068 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c81137c96b254afdc7a6524989e0eabb780924be24e268e7e3d8270a5fd64571\": container with ID starting with c81137c96b254afdc7a6524989e0eabb780924be24e268e7e3d8270a5fd64571 not found: ID does not exist" containerID="c81137c96b254afdc7a6524989e0eabb780924be24e268e7e3d8270a5fd64571" Jul 31 19:42:47.920128 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.920092 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c81137c96b254afdc7a6524989e0eabb780924be24e268e7e3d8270a5fd64571"} err="failed to get container status \"c81137c96b254afdc7a6524989e0eabb780924be24e268e7e3d8270a5fd64571\": rpc error: code = NotFound desc = could not find container \"c81137c96b254afdc7a6524989e0eabb780924be24e268e7e3d8270a5fd64571\": container with ID starting with c81137c96b254afdc7a6524989e0eabb780924be24e268e7e3d8270a5fd64571 not found: ID does not exist" Jul 31 19:42:47.999500 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.999465 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f0b2bc3-d509-461c-90a5-0d68eb70e012-istio-envoy\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:42:47.999500 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.999492 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f0b2bc3-d509-461c-90a5-0d68eb70e012-workload-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:42:47.999500 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.999501 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f0b2bc3-d509-461c-90a5-0d68eb70e012-istio-podinfo\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:42:47.999500 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.999510 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzhv7\" (UniqueName: \"kubernetes.io/projected/3f0b2bc3-d509-461c-90a5-0d68eb70e012-kube-api-access-kzhv7\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:42:47.999770 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:47.999519 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f0b2bc3-d509-461c-90a5-0d68eb70e012-istio-token\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:42:48.219543 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:48.219509 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw"] Jul 31 19:42:48.226926 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:48.226897 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-fd0284-openshift-default-5677c847cb-nnwvw"] Jul 31 19:42:49.889979 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:49.889124 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f0b2bc3-d509-461c-90a5-0d68eb70e012" path="/var/lib/kubelet/pods/3f0b2bc3-d509-461c-90a5-0d68eb70e012/volumes" Jul 31 19:42:53.477702 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:53.477666 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:42:53.541012 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:53.540975 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:42:53.601541 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:53.601511 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:42:53.672053 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:53.672005 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:42:54.072265 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:54.072204 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:42:55.651210 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:42:55.651173 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-h7vrb"] Jul 31 19:43:00.285139 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:00.285098 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:43:00.342835 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:00.342802 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:43:00.541401 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:00.541322 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2"] Jul 31 19:43:00.541594 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:00.541567 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" podUID="edcd39a9-332e-42bf-b284-a49014964bb6" containerName="istio-proxy" containerID="cri-o://507ebc11edecbb79291f73ab453efee9393293731bd9f3b90b5382acd16c0f99" gracePeriod=30 Jul 31 19:43:00.578295 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:00.578227 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:43:00.761779 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:00.761742 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:43:01.178064 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:01.178036 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9bd55985b-2p6jj_8b24f01a-7f75-43e1-9a87-631a7a69eca8/authorino/0.log" Jul 31 19:43:04.836785 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:04.836743 2565 patch_prober.go:28] interesting pod/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 31 19:43:04.837156 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:04.836815 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" podUID="edcd39a9-332e-42bf-b284-a49014964bb6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 31 19:43:05.799517 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:05.799492 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:43:05.968911 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:05.968814 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcd39a9-332e-42bf-b284-a49014964bb6-credential-socket\") pod \"edcd39a9-332e-42bf-b284-a49014964bb6\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " Jul 31 19:43:05.968911 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:05.968908 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edcd39a9-332e-42bf-b284-a49014964bb6-istio-token\") pod \"edcd39a9-332e-42bf-b284-a49014964bb6\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " Jul 31 19:43:05.969480 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:05.968953 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edcd39a9-332e-42bf-b284-a49014964bb6-istio-podinfo\") pod \"edcd39a9-332e-42bf-b284-a49014964bb6\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " Jul 31 19:43:05.969480 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:05.968978 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edcd39a9-332e-42bf-b284-a49014964bb6-istio-data\") pod \"edcd39a9-332e-42bf-b284-a49014964bb6\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " Jul 31 19:43:05.969480 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:05.969001 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edcd39a9-332e-42bf-b284-a49014964bb6-istiod-ca-cert\") pod \"edcd39a9-332e-42bf-b284-a49014964bb6\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " Jul 31 19:43:05.969480 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:05.969033 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edcd39a9-332e-42bf-b284-a49014964bb6-workload-certs\") pod \"edcd39a9-332e-42bf-b284-a49014964bb6\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " Jul 31 19:43:05.969480 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:05.969084 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22z42\" (UniqueName: \"kubernetes.io/projected/edcd39a9-332e-42bf-b284-a49014964bb6-kube-api-access-22z42\") pod \"edcd39a9-332e-42bf-b284-a49014964bb6\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " Jul 31 19:43:05.969480 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:05.969104 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcd39a9-332e-42bf-b284-a49014964bb6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "edcd39a9-332e-42bf-b284-a49014964bb6" (UID: "edcd39a9-332e-42bf-b284-a49014964bb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:43:05.969480 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:05.969126 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcd39a9-332e-42bf-b284-a49014964bb6-workload-socket\") pod \"edcd39a9-332e-42bf-b284-a49014964bb6\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " Jul 31 19:43:05.969480 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:05.969180 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edcd39a9-332e-42bf-b284-a49014964bb6-istio-envoy\") pod \"edcd39a9-332e-42bf-b284-a49014964bb6\" (UID: \"edcd39a9-332e-42bf-b284-a49014964bb6\") " Jul 31 19:43:05.969480 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:05.969426 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcd39a9-332e-42bf-b284-a49014964bb6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "edcd39a9-332e-42bf-b284-a49014964bb6" (UID: "edcd39a9-332e-42bf-b284-a49014964bb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:43:05.969913 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:05.969632 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcd39a9-332e-42bf-b284-a49014964bb6-istio-data" (OuterVolumeSpecName: "istio-data") pod "edcd39a9-332e-42bf-b284-a49014964bb6" (UID: "edcd39a9-332e-42bf-b284-a49014964bb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:43:05.969913 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:05.969641 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edcd39a9-332e-42bf-b284-a49014964bb6-workload-certs\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:43:05.969913 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:05.969677 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcd39a9-332e-42bf-b284-a49014964bb6-credential-socket\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:43:05.969913 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:05.969722 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcd39a9-332e-42bf-b284-a49014964bb6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "edcd39a9-332e-42bf-b284-a49014964bb6" (UID: "edcd39a9-332e-42bf-b284-a49014964bb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:43:05.969913 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:05.969845 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edcd39a9-332e-42bf-b284-a49014964bb6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "edcd39a9-332e-42bf-b284-a49014964bb6" (UID: "edcd39a9-332e-42bf-b284-a49014964bb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 19:43:05.971147 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:05.971115 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edcd39a9-332e-42bf-b284-a49014964bb6-istio-token" (OuterVolumeSpecName: "istio-token") pod "edcd39a9-332e-42bf-b284-a49014964bb6" (UID: "edcd39a9-332e-42bf-b284-a49014964bb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:43:05.971284 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:05.971168 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/edcd39a9-332e-42bf-b284-a49014964bb6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "edcd39a9-332e-42bf-b284-a49014964bb6" (UID: "edcd39a9-332e-42bf-b284-a49014964bb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 19:43:05.971662 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:05.971635 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcd39a9-332e-42bf-b284-a49014964bb6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "edcd39a9-332e-42bf-b284-a49014964bb6" (UID: "edcd39a9-332e-42bf-b284-a49014964bb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:43:05.971748 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:05.971692 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edcd39a9-332e-42bf-b284-a49014964bb6-kube-api-access-22z42" (OuterVolumeSpecName: "kube-api-access-22z42") pod "edcd39a9-332e-42bf-b284-a49014964bb6" (UID: "edcd39a9-332e-42bf-b284-a49014964bb6"). InnerVolumeSpecName "kube-api-access-22z42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:43:05.982156 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:05.982128 2565 generic.go:356] "Generic (PLEG): container finished" podID="edcd39a9-332e-42bf-b284-a49014964bb6" containerID="507ebc11edecbb79291f73ab453efee9393293731bd9f3b90b5382acd16c0f99" exitCode=0 Jul 31 19:43:05.982277 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:05.982175 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" Jul 31 19:43:05.982277 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:05.982205 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" event={"ID":"edcd39a9-332e-42bf-b284-a49014964bb6","Type":"ContainerDied","Data":"507ebc11edecbb79291f73ab453efee9393293731bd9f3b90b5382acd16c0f99"} Jul 31 19:43:05.982386 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:05.982280 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2" event={"ID":"edcd39a9-332e-42bf-b284-a49014964bb6","Type":"ContainerDied","Data":"ac8a9144135672a258470c7bb6f69d874a56033cbc7e1cbf56f1a9eedc3accfc"} Jul 31 19:43:05.982386 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:05.982313 2565 scope.go:117] "RemoveContainer" containerID="507ebc11edecbb79291f73ab453efee9393293731bd9f3b90b5382acd16c0f99" Jul 31 19:43:05.992159 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:05.992142 2565 scope.go:117] "RemoveContainer" containerID="6d517e32997a47a436d5be6a8183a4b609c330d2eed44d02d95c8b7b229f15f2" Jul 31 19:43:06.000044 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:06.000020 2565 scope.go:117] "RemoveContainer" containerID="507ebc11edecbb79291f73ab453efee9393293731bd9f3b90b5382acd16c0f99" Jul 31 19:43:06.000300 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:43:06.000281 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"507ebc11edecbb79291f73ab453efee9393293731bd9f3b90b5382acd16c0f99\": container with ID starting with 507ebc11edecbb79291f73ab453efee9393293731bd9f3b90b5382acd16c0f99 not found: ID does not exist" containerID="507ebc11edecbb79291f73ab453efee9393293731bd9f3b90b5382acd16c0f99" Jul 31 19:43:06.000375 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:06.000312 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"507ebc11edecbb79291f73ab453efee9393293731bd9f3b90b5382acd16c0f99"} err="failed to get container status \"507ebc11edecbb79291f73ab453efee9393293731bd9f3b90b5382acd16c0f99\": rpc error: code = NotFound desc = could not find container \"507ebc11edecbb79291f73ab453efee9393293731bd9f3b90b5382acd16c0f99\": container with ID starting with 507ebc11edecbb79291f73ab453efee9393293731bd9f3b90b5382acd16c0f99 not found: ID does not exist" Jul 31 19:43:06.000375 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:06.000337 2565 scope.go:117] "RemoveContainer" containerID="6d517e32997a47a436d5be6a8183a4b609c330d2eed44d02d95c8b7b229f15f2" Jul 31 19:43:06.000611 ip-10-0-139-39 kubenswrapper[2565]: E0731 19:43:06.000593 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d517e32997a47a436d5be6a8183a4b609c330d2eed44d02d95c8b7b229f15f2\": container with ID starting with 6d517e32997a47a436d5be6a8183a4b609c330d2eed44d02d95c8b7b229f15f2 not found: ID does not exist" containerID="6d517e32997a47a436d5be6a8183a4b609c330d2eed44d02d95c8b7b229f15f2" Jul 31 19:43:06.000661 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:06.000617 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d517e32997a47a436d5be6a8183a4b609c330d2eed44d02d95c8b7b229f15f2"} err="failed to get container status \"6d517e32997a47a436d5be6a8183a4b609c330d2eed44d02d95c8b7b229f15f2\": rpc error: code = NotFound desc = could not find container \"6d517e32997a47a436d5be6a8183a4b609c330d2eed44d02d95c8b7b229f15f2\": container with ID starting with 6d517e32997a47a436d5be6a8183a4b609c330d2eed44d02d95c8b7b229f15f2 not found: ID does not exist" Jul 31 19:43:06.010220 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:06.010198 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2"] Jul 31 19:43:06.024369 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:06.024347 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-09a154-openshift-default-6f5586bb5b-9rxb2"] Jul 31 19:43:06.070787 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:06.070759 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcd39a9-332e-42bf-b284-a49014964bb6-workload-socket\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:43:06.070787 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:06.070780 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edcd39a9-332e-42bf-b284-a49014964bb6-istio-envoy\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:43:06.070935 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:06.070794 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edcd39a9-332e-42bf-b284-a49014964bb6-istio-token\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:43:06.070935 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:06.070803 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edcd39a9-332e-42bf-b284-a49014964bb6-istio-podinfo\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:43:06.070935 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:06.070811 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edcd39a9-332e-42bf-b284-a49014964bb6-istio-data\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:43:06.070935 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:06.070818 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edcd39a9-332e-42bf-b284-a49014964bb6-istiod-ca-cert\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:43:06.070935 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:06.070826 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22z42\" (UniqueName: \"kubernetes.io/projected/edcd39a9-332e-42bf-b284-a49014964bb6-kube-api-access-22z42\") on node \"ip-10-0-139-39.ec2.internal\" DevicePath \"\"" Jul 31 19:43:06.185668 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:06.185637 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-bpgf8_eb861fcc-08c9-427e-8a9a-bd1d9bb8c607/manager/0.log" Jul 31 19:43:06.304192 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:06.304155 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-9tbcv_73c1838d-6c05-4576-8f70-b23a70a55d4f/manager/0.log" Jul 31 19:43:06.545858 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:06.545833 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-wnvkm_bec16275-6e1a-4d0e-84a8-b648f317611a/server/0.log" Jul 31 19:43:06.657702 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:06.657616 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-ktlkc_2b6c0760-f3ae-46b2-911a-c1e7e2663485/manager/0.log" Jul 31 19:43:06.762839 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:06.762811 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84c85c7ff9-72wp8_9eca874d-935c-4e8f-a053-dd892772587f/copy-manifests/0.log" Jul 31 19:43:06.769215 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:06.769185 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84c85c7ff9-72wp8_9eca874d-935c-4e8f-a053-dd892772587f/manager/0.log" Jul 31 19:43:07.887776 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:07.887745 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edcd39a9-332e-42bf-b284-a49014964bb6" path="/var/lib/kubelet/pods/edcd39a9-332e-42bf-b284-a49014964bb6/volumes" Jul 31 19:43:08.533532 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:08.533497 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9bd55985b-2p6jj_8b24f01a-7f75-43e1-9a87-631a7a69eca8/authorino/0.log" Jul 31 19:43:08.645711 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:08.645673 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-4z7w6_fb30c671-c6c4-4301-9a08-0a72025c1fc5/manager/0.log" Jul 31 19:43:08.748336 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:08.748308 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-5k7rm_d4fcf557-31a9-4bd9-acdd-6b12028632f2/manager/0.log" Jul 31 19:43:08.851942 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:08.851858 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-9fgbg_7398c822-ac51-47b6-acef-cb622d693145/kuadrant-console-plugin/0.log" Jul 31 19:43:09.078284 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:09.078230 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-h7vrb_90363ad8-673b-45b6-887f-c82f70632eba/limitador/0.log" Jul 31 19:43:09.515590 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:09.515557 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d_eb55cf00-4290-49f4-8367-1d2d69a90016/istio-proxy/0.log" Jul 31 19:43:09.744857 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:09.744820 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86dbc777-nbl8z_1b56a1d6-fe42-4319-8e31-90bc4def9285/kube-auth-proxy/0.log" Jul 31 19:43:09.973150 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:09.973071 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-w8k4k_ed4e23e4-7b4d-4214-a571-0f54d841c7c5/istio-proxy/1.log" Jul 31 19:43:10.312887 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:10.312858 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-685dfffc96-vrmgp_a8c11319-084e-4318-9e92-5a4a68d8c916/router/0.log" Jul 31 19:43:10.959462 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:10.959432 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmzld4_10caf125-874d-4caf-bbfc-0d196da1008b/main/0.log" Jul 31 19:43:11.175561 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:11.175533 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-pnhgv_cc47d375-9655-47db-8feb-e96c91236d91/main/0.log" Jul 31 19:43:16.586615 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.586577 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-x2d7j/must-gather-2kcjx"] Jul 31 19:43:16.586986 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.586923 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54fff827-a0b0-4999-9f01-7d438d3270f4" containerName="istio-proxy" Jul 31 19:43:16.586986 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.586934 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="54fff827-a0b0-4999-9f01-7d438d3270f4" containerName="istio-proxy" Jul 31 19:43:16.586986 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.586943 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f0b2bc3-d509-461c-90a5-0d68eb70e012" containerName="istio-proxy" Jul 31 19:43:16.586986 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.586949 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f0b2bc3-d509-461c-90a5-0d68eb70e012" containerName="istio-proxy" Jul 31 19:43:16.586986 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.586957 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a" containerName="istio-proxy" Jul 31 19:43:16.586986 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.586963 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a" containerName="istio-proxy" Jul 31 19:43:16.586986 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.586973 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f0b2bc3-d509-461c-90a5-0d68eb70e012" containerName="istio-proxy" Jul 31 19:43:16.586986 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.586978 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f0b2bc3-d509-461c-90a5-0d68eb70e012" containerName="istio-proxy" Jul 31 19:43:16.586986 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.586986 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edcd39a9-332e-42bf-b284-a49014964bb6" containerName="istio-proxy" Jul 31 19:43:16.586986 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.586991 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="edcd39a9-332e-42bf-b284-a49014964bb6" containerName="istio-proxy" Jul 31 19:43:16.587318 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.586999 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edcd39a9-332e-42bf-b284-a49014964bb6" containerName="istio-proxy" Jul 31 19:43:16.587318 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.587004 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="edcd39a9-332e-42bf-b284-a49014964bb6" containerName="istio-proxy" Jul 31 19:43:16.587318 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.587011 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54fff827-a0b0-4999-9f01-7d438d3270f4" containerName="istio-proxy" Jul 31 19:43:16.587318 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.587017 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="54fff827-a0b0-4999-9f01-7d438d3270f4" containerName="istio-proxy" Jul 31 19:43:16.587318 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.587029 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbe41a06-fe16-4d89-b1f1-d98c6baab2a6" containerName="istio-proxy" Jul 31 19:43:16.587318 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.587034 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbe41a06-fe16-4d89-b1f1-d98c6baab2a6" containerName="istio-proxy" Jul 31 19:43:16.587318 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.587104 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a" containerName="istio-proxy" Jul 31 19:43:16.587318 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.587111 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="edcd39a9-332e-42bf-b284-a49014964bb6" containerName="istio-proxy" Jul 31 19:43:16.587318 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.587118 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="54fff827-a0b0-4999-9f01-7d438d3270f4" containerName="istio-proxy" Jul 31 19:43:16.587318 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.587124 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a" containerName="istio-proxy" Jul 31 19:43:16.587318 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.587130 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbe41a06-fe16-4d89-b1f1-d98c6baab2a6" containerName="istio-proxy" Jul 31 19:43:16.587318 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.587135 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="edcd39a9-332e-42bf-b284-a49014964bb6" containerName="istio-proxy" Jul 31 19:43:16.587318 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.587141 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f0b2bc3-d509-461c-90a5-0d68eb70e012" containerName="istio-proxy" Jul 31 19:43:16.587318 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.587146 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="54fff827-a0b0-4999-9f01-7d438d3270f4" containerName="istio-proxy" Jul 31 19:43:16.587318 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.587202 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbe41a06-fe16-4d89-b1f1-d98c6baab2a6" containerName="istio-proxy" Jul 31 19:43:16.587318 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.587207 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbe41a06-fe16-4d89-b1f1-d98c6baab2a6" containerName="istio-proxy" Jul 31 19:43:16.587318 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.587219 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a" containerName="istio-proxy" Jul 31 19:43:16.587318 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.587226 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e21f9ca4-b9a2-4470-b6c6-58b81cf76a2a" containerName="istio-proxy" Jul 31 19:43:16.587318 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.587293 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbe41a06-fe16-4d89-b1f1-d98c6baab2a6" containerName="istio-proxy" Jul 31 19:43:16.587318 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.587301 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f0b2bc3-d509-461c-90a5-0d68eb70e012" containerName="istio-proxy" Jul 31 19:43:16.591993 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.591976 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x2d7j/must-gather-2kcjx" Jul 31 19:43:16.594152 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.594129 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x2d7j\"/\"kube-root-ca.crt\"" Jul 31 19:43:16.594277 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.594138 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x2d7j\"/\"openshift-service-ca.crt\"" Jul 31 19:43:16.594277 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.594204 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-x2d7j\"/\"default-dockercfg-kfhf6\"" Jul 31 19:43:16.599324 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.599302 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x2d7j/must-gather-2kcjx"] Jul 31 19:43:16.604948 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.604922 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:43:16.670579 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.670550 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/74ac4ea1-1de1-48f7-8ad8-b95023293c4a-must-gather-output\") pod \"must-gather-2kcjx\" (UID: \"74ac4ea1-1de1-48f7-8ad8-b95023293c4a\") " pod="openshift-must-gather-x2d7j/must-gather-2kcjx" Jul 31 19:43:16.670741 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.670614 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbblh\" (UniqueName: \"kubernetes.io/projected/74ac4ea1-1de1-48f7-8ad8-b95023293c4a-kube-api-access-gbblh\") pod \"must-gather-2kcjx\" (UID: \"74ac4ea1-1de1-48f7-8ad8-b95023293c4a\") " pod="openshift-must-gather-x2d7j/must-gather-2kcjx" Jul 31 19:43:16.679720 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.679690 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:43:16.772027 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.771992 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/74ac4ea1-1de1-48f7-8ad8-b95023293c4a-must-gather-output\") pod \"must-gather-2kcjx\" (UID: \"74ac4ea1-1de1-48f7-8ad8-b95023293c4a\") " pod="openshift-must-gather-x2d7j/must-gather-2kcjx" Jul 31 19:43:16.772183 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.772058 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbblh\" (UniqueName: \"kubernetes.io/projected/74ac4ea1-1de1-48f7-8ad8-b95023293c4a-kube-api-access-gbblh\") pod \"must-gather-2kcjx\" (UID: \"74ac4ea1-1de1-48f7-8ad8-b95023293c4a\") " pod="openshift-must-gather-x2d7j/must-gather-2kcjx" Jul 31 19:43:16.772371 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.772352 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/74ac4ea1-1de1-48f7-8ad8-b95023293c4a-must-gather-output\") pod \"must-gather-2kcjx\" (UID: \"74ac4ea1-1de1-48f7-8ad8-b95023293c4a\") " pod="openshift-must-gather-x2d7j/must-gather-2kcjx" Jul 31 19:43:16.780852 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.780828 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbblh\" (UniqueName: \"kubernetes.io/projected/74ac4ea1-1de1-48f7-8ad8-b95023293c4a-kube-api-access-gbblh\") pod \"must-gather-2kcjx\" (UID: \"74ac4ea1-1de1-48f7-8ad8-b95023293c4a\") " pod="openshift-must-gather-x2d7j/must-gather-2kcjx" Jul 31 19:43:16.901966 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:16.901874 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x2d7j/must-gather-2kcjx" Jul 31 19:43:17.039675 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:17.039609 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x2d7j/must-gather-2kcjx"] Jul 31 19:43:17.041941 ip-10-0-139-39 kubenswrapper[2565]: W0731 19:43:17.041916 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74ac4ea1_1de1_48f7_8ad8_b95023293c4a.slice/crio-cea1da4cc3dc36bad2711b31ebbb05ed1c9dde642d477107ff64ecd95bdbabb4 WatchSource:0}: Error finding container cea1da4cc3dc36bad2711b31ebbb05ed1c9dde642d477107ff64ecd95bdbabb4: Status 404 returned error can't find the container with id cea1da4cc3dc36bad2711b31ebbb05ed1c9dde642d477107ff64ecd95bdbabb4 Jul 31 19:43:17.043658 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:17.043642 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 19:43:18.046128 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:18.046097 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x2d7j/must-gather-2kcjx" event={"ID":"74ac4ea1-1de1-48f7-8ad8-b95023293c4a","Type":"ContainerStarted","Data":"430d43483ea8b7e67e18f453b92ce8e9e642c1aad467e07de2cc2229f7833a9a"} Jul 31 19:43:18.046463 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:18.046150 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x2d7j/must-gather-2kcjx" event={"ID":"74ac4ea1-1de1-48f7-8ad8-b95023293c4a","Type":"ContainerStarted","Data":"5785d1e477e5688dfe4827d3c530005b0dda2ee217ee949c7d80f439e0d2d7ec"} Jul 31 19:43:18.046463 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:18.046161 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x2d7j/must-gather-2kcjx" event={"ID":"74ac4ea1-1de1-48f7-8ad8-b95023293c4a","Type":"ContainerStarted","Data":"cea1da4cc3dc36bad2711b31ebbb05ed1c9dde642d477107ff64ecd95bdbabb4"} Jul 31 19:43:19.071692 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:19.071642 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-x2d7j/must-gather-2kcjx" podStartSLOduration=2.303221769 podStartE2EDuration="3.071625698s" totalImagesPullingTime="768.403929ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:43:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:43:19.0690481 +0000 UTC m=+3575.707037902" watchObservedRunningTime="2026-07-31 19:43:19.071625698 +0000 UTC m=+3575.709615536" Jul 31 19:43:19.466082 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:19.466000 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2rkhw_e8223854-5549-4701-b1ce-506ed19d8abe/global-pull-secret-syncer/0.log" Jul 31 19:43:19.617556 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:19.617521 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" Jul 31 19:43:19.752446 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:19.752412 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-39.ec2.internal_6781281007e7ddfaa5111df980719fef/haproxy/0.log" Jul 31 19:43:24.930256 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:24.928663 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9bd55985b-2p6jj_8b24f01a-7f75-43e1-9a87-631a7a69eca8/authorino/0.log" Jul 31 19:43:24.959310 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:24.959271 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-4z7w6_fb30c671-c6c4-4301-9a08-0a72025c1fc5/manager/0.log" Jul 31 19:43:24.994488 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:24.994417 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-5k7rm_d4fcf557-31a9-4bd9-acdd-6b12028632f2/manager/0.log" Jul 31 19:43:25.021778 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:25.021748 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-9fgbg_7398c822-ac51-47b6-acef-cb622d693145/kuadrant-console-plugin/0.log" Jul 31 19:43:25.193044 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:25.192964 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-h7vrb_90363ad8-673b-45b6-887f-c82f70632eba/limitador/0.log" Jul 31 19:43:26.992962 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:26.992883 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-m76n7_45c0e651-aba9-4741-84db-8a06cd82c46e/cluster-monitoring-operator/0.log" Jul 31 19:43:27.267520 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:27.267489 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-474bp_2315e49b-1ea3-4d46-abe6-9182a23ba0c4/node-exporter/0.log" Jul 31 19:43:27.291334 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:27.291300 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-474bp_2315e49b-1ea3-4d46-abe6-9182a23ba0c4/kube-rbac-proxy/0.log" Jul 31 19:43:27.319196 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:27.319166 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-474bp_2315e49b-1ea3-4d46-abe6-9182a23ba0c4/init-textfile/0.log" Jul 31 19:43:27.855878 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:27.855842 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:43:27.912604 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:27.912558 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7s5zf_ffecd04e-7d64-429f-95a8-20647557526d/konnectivity-agent/0.log" limit=1024 Jul 31 19:43:29.190503 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:29.190475 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-f77f48c6d-6fcww_0b029477-8e2e-46ba-8209-d3cd2f49f582/networking-console-plugin/0.log" Jul 31 19:43:29.546986 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:29.546951 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-x2d7j/perf-node-gather-daemonset-x7xwr"] Jul 31 19:43:29.553384 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:29.552318 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-x7xwr" Jul 31 19:43:29.560827 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:29.560732 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x2d7j/perf-node-gather-daemonset-x7xwr"] Jul 31 19:43:29.627420 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:29.627369 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07d439a7-a60a-4347-a49a-d3962160f8af-sys\") pod \"perf-node-gather-daemonset-x7xwr\" (UID: \"07d439a7-a60a-4347-a49a-d3962160f8af\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-x7xwr" Jul 31 19:43:29.627721 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:29.627678 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v87x8\" (UniqueName: \"kubernetes.io/projected/07d439a7-a60a-4347-a49a-d3962160f8af-kube-api-access-v87x8\") pod \"perf-node-gather-daemonset-x7xwr\" (UID: \"07d439a7-a60a-4347-a49a-d3962160f8af\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-x7xwr" Jul 31 19:43:29.627887 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:29.627871 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07d439a7-a60a-4347-a49a-d3962160f8af-podres\") pod \"perf-node-gather-daemonset-x7xwr\" (UID: \"07d439a7-a60a-4347-a49a-d3962160f8af\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-x7xwr" Jul 31 19:43:29.628077 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:29.628049 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07d439a7-a60a-4347-a49a-d3962160f8af-proc\") pod \"perf-node-gather-daemonset-x7xwr\" (UID: \"07d439a7-a60a-4347-a49a-d3962160f8af\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-x7xwr" Jul 31 19:43:29.628252 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:29.628219 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07d439a7-a60a-4347-a49a-d3962160f8af-lib-modules\") pod \"perf-node-gather-daemonset-x7xwr\" (UID: \"07d439a7-a60a-4347-a49a-d3962160f8af\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-x7xwr" Jul 31 19:43:29.729146 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:29.729104 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07d439a7-a60a-4347-a49a-d3962160f8af-lib-modules\") pod \"perf-node-gather-daemonset-x7xwr\" (UID: \"07d439a7-a60a-4347-a49a-d3962160f8af\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-x7xwr" Jul 31 19:43:29.729337 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:29.729208 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07d439a7-a60a-4347-a49a-d3962160f8af-sys\") pod \"perf-node-gather-daemonset-x7xwr\" (UID: \"07d439a7-a60a-4347-a49a-d3962160f8af\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-x7xwr" Jul 31 19:43:29.729337 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:29.729278 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v87x8\" (UniqueName: \"kubernetes.io/projected/07d439a7-a60a-4347-a49a-d3962160f8af-kube-api-access-v87x8\") pod \"perf-node-gather-daemonset-x7xwr\" (UID: \"07d439a7-a60a-4347-a49a-d3962160f8af\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-x7xwr" Jul 31 19:43:29.729337 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:29.729317 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07d439a7-a60a-4347-a49a-d3962160f8af-podres\") pod \"perf-node-gather-daemonset-x7xwr\" (UID: \"07d439a7-a60a-4347-a49a-d3962160f8af\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-x7xwr" Jul 31 19:43:29.729479 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:29.729332 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07d439a7-a60a-4347-a49a-d3962160f8af-sys\") pod \"perf-node-gather-daemonset-x7xwr\" (UID: \"07d439a7-a60a-4347-a49a-d3962160f8af\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-x7xwr" Jul 31 19:43:29.729479 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:29.729332 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07d439a7-a60a-4347-a49a-d3962160f8af-lib-modules\") pod \"perf-node-gather-daemonset-x7xwr\" (UID: \"07d439a7-a60a-4347-a49a-d3962160f8af\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-x7xwr" Jul 31 19:43:29.729479 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:29.729470 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07d439a7-a60a-4347-a49a-d3962160f8af-proc\") pod \"perf-node-gather-daemonset-x7xwr\" (UID: \"07d439a7-a60a-4347-a49a-d3962160f8af\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-x7xwr" Jul 31 19:43:29.729588 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:29.729541 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07d439a7-a60a-4347-a49a-d3962160f8af-podres\") pod \"perf-node-gather-daemonset-x7xwr\" (UID: \"07d439a7-a60a-4347-a49a-d3962160f8af\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-x7xwr" Jul 31 19:43:29.729588 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:29.729566 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07d439a7-a60a-4347-a49a-d3962160f8af-proc\") pod \"perf-node-gather-daemonset-x7xwr\" (UID: \"07d439a7-a60a-4347-a49a-d3962160f8af\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-x7xwr" Jul 31 19:43:29.740711 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:29.740681 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v87x8\" (UniqueName: \"kubernetes.io/projected/07d439a7-a60a-4347-a49a-d3962160f8af-kube-api-access-v87x8\") pod \"perf-node-gather-daemonset-x7xwr\" (UID: \"07d439a7-a60a-4347-a49a-d3962160f8af\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-x7xwr" Jul 31 19:43:29.866821 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:29.866736 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-x7xwr" Jul 31 19:43:30.083521 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:30.082093 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x2d7j/perf-node-gather-daemonset-x7xwr"] Jul 31 19:43:30.091964 ip-10-0-139-39 kubenswrapper[2565]: W0731 19:43:30.091927 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod07d439a7_a60a_4347_a49a_d3962160f8af.slice/crio-458e8fc8235c316fbf4e76af610fa7073b539f62608fb43e8e3fa9e579b1cbc9 WatchSource:0}: Error finding container 458e8fc8235c316fbf4e76af610fa7073b539f62608fb43e8e3fa9e579b1cbc9: Status 404 returned error can't find the container with id 458e8fc8235c316fbf4e76af610fa7073b539f62608fb43e8e3fa9e579b1cbc9 Jul 31 19:43:30.140970 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:30.140925 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-x7xwr" event={"ID":"07d439a7-a60a-4347-a49a-d3962160f8af","Type":"ContainerStarted","Data":"458e8fc8235c316fbf4e76af610fa7073b539f62608fb43e8e3fa9e579b1cbc9"} Jul 31 19:43:30.499844 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:30.499801 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5c8df6ddb9-mzdgl_e5b8c507-5033-476e-8c9f-dd129e98e2b0/console/0.log" Jul 31 19:43:30.543378 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:30.543344 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-77c974ffcf-hb2hk_e3d33d0d-4656-4016-be66-99d9aa6e0994/download-server/0.log" Jul 31 19:43:31.149074 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:31.149037 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-x7xwr" event={"ID":"07d439a7-a60a-4347-a49a-d3962160f8af","Type":"ContainerStarted","Data":"2b43d41cedad28a98368ed789a907ff24ef4c014762c6dd9df75b13950137fc1"} Jul 31 19:43:31.149411 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:31.149392 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-x7xwr" Jul 31 19:43:31.166149 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:31.166095 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-x7xwr" podStartSLOduration=2.166075328 podStartE2EDuration="2.166075328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 19:43:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:43:31.165335003 +0000 UTC m=+3587.803324804" watchObservedRunningTime="2026-07-31 19:43:31.166075328 +0000 UTC m=+3587.804065130" Jul 31 19:43:32.461933 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:32.461899 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dgmn6_98250d35-1d7d-4f70-8350-b7865456ae46/dns/0.log" Jul 31 19:43:32.486431 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:32.486405 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dgmn6_98250d35-1d7d-4f70-8350-b7865456ae46/kube-rbac-proxy/0.log" Jul 31 19:43:32.538022 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:32.537994 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fjbvh_ba85de76-d297-4a7b-bb30-9c12572d9811/dns-node-resolver/0.log" Jul 31 19:43:33.132304 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:33.132267 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-b4695766f-dvcj6_3bea09ae-5eb5-4b26-a6d3-793ed2defb45/registry/0.log" Jul 31 19:43:33.186821 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:33.186762 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sbmpv_fa723eb5-7c9f-4676-aeda-50b07733ff96/node-ca/0.log" Jul 31 19:43:34.191607 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:34.191568 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d_eb55cf00-4290-49f4-8367-1d2d69a90016/istio-proxy/0.log" Jul 31 19:43:34.379937 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:34.379900 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86dbc777-nbl8z_1b56a1d6-fe42-4319-8e31-90bc4def9285/kube-auth-proxy/0.log" Jul 31 19:43:34.437520 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:34.437490 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-w8k4k_ed4e23e4-7b4d-4214-a571-0f54d841c7c5/istio-proxy/0.log" Jul 31 19:43:34.447254 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:34.447153 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-w8k4k_ed4e23e4-7b4d-4214-a571-0f54d841c7c5/istio-proxy/1.log" Jul 31 19:43:34.555426 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:34.555383 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-685dfffc96-vrmgp_a8c11319-084e-4318-9e92-5a4a68d8c916/router/0.log" Jul 31 19:43:35.419778 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:35.419743 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9kv6f_9bd58afe-8ba9-4f95-ae27-c97586229a7a/serve-healthcheck-canary/0.log" Jul 31 19:43:35.945480 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:35.945441 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-rkrsg_24f12167-7a95-45d7-bc34-9cafeec2201e/insights-operator/0.log" Jul 31 19:43:35.948062 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:35.948038 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-rkrsg_24f12167-7a95-45d7-bc34-9cafeec2201e/insights-operator/1.log" Jul 31 19:43:36.151525 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:36.151497 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vrdbp_0e664160-fd29-4f67-b32e-8ae37b2641be/kube-rbac-proxy/0.log" Jul 31 19:43:36.175259 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:36.175210 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vrdbp_0e664160-fd29-4f67-b32e-8ae37b2641be/exporter/0.log" Jul 31 19:43:36.202502 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:36.202432 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vrdbp_0e664160-fd29-4f67-b32e-8ae37b2641be/extractor/0.log" Jul 31 19:43:38.177351 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:38.177317 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-x7xwr" Jul 31 19:43:38.357827 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:38.357797 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-bpgf8_eb861fcc-08c9-427e-8a9a-bd1d9bb8c607/manager/0.log" Jul 31 19:43:38.666673 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:38.666639 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-9tbcv_73c1838d-6c05-4576-8f70-b23a70a55d4f/manager/0.log" Jul 31 19:43:38.976296 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:38.976200 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-wnvkm_bec16275-6e1a-4d0e-84a8-b648f317611a/server/0.log" Jul 31 19:43:39.075975 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:39.075945 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-ktlkc_2b6c0760-f3ae-46b2-911a-c1e7e2663485/manager/0.log" Jul 31 19:43:39.102923 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:39.102878 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84c85c7ff9-72wp8_9eca874d-935c-4e8f-a053-dd892772587f/manager/0.log" Jul 31 19:43:39.130792 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:39.130763 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84c85c7ff9-72wp8_9eca874d-935c-4e8f-a053-dd892772587f/copy-manifests/0.log" Jul 31 19:43:40.001916 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:40.001883 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d_eb55cf00-4290-49f4-8367-1d2d69a90016/istio-proxy/0.log" Jul 31 19:43:40.206405 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:40.206376 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86dbc777-nbl8z_1b56a1d6-fe42-4319-8e31-90bc4def9285/kube-auth-proxy/0.log" Jul 31 19:43:40.244980 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:40.244886 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-w8k4k_ed4e23e4-7b4d-4214-a571-0f54d841c7c5/istio-proxy/0.log" Jul 31 19:43:40.255275 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:40.255222 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-w8k4k_ed4e23e4-7b4d-4214-a571-0f54d841c7c5/istio-proxy/1.log" Jul 31 19:43:40.332093 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:40.332067 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-685dfffc96-vrmgp_a8c11319-084e-4318-9e92-5a4a68d8c916/router/0.log" Jul 31 19:43:40.933226 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:40.933143 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-7kspr_69612ccc-85d1-4b6e-964e-6aa55d91a327/openshift-lws-operator/0.log" Jul 31 19:43:41.810248 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:41.810203 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c9pb8d_eb55cf00-4290-49f4-8367-1d2d69a90016/istio-proxy/0.log" Jul 31 19:43:41.989664 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:41.989638 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86dbc777-nbl8z_1b56a1d6-fe42-4319-8e31-90bc4def9285/kube-auth-proxy/0.log" Jul 31 19:43:42.030130 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:42.030078 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-w8k4k_ed4e23e4-7b4d-4214-a571-0f54d841c7c5/istio-proxy/0.log" Jul 31 19:43:42.038802 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:42.038766 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-w8k4k_ed4e23e4-7b4d-4214-a571-0f54d841c7c5/istio-proxy/1.log" Jul 31 19:43:42.121447 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:42.121407 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-685dfffc96-vrmgp_a8c11319-084e-4318-9e92-5a4a68d8c916/router/0.log" Jul 31 19:43:48.042226 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:48.042199 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p9mqb_cdddb050-55be-48e1-9a8e-df26711e4416/kube-multus-additional-cni-plugins/0.log" Jul 31 19:43:48.068543 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:48.068516 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p9mqb_cdddb050-55be-48e1-9a8e-df26711e4416/egress-router-binary-copy/0.log" Jul 31 19:43:48.092934 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:48.092904 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p9mqb_cdddb050-55be-48e1-9a8e-df26711e4416/cni-plugins/0.log" Jul 31 19:43:48.115114 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:48.115087 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p9mqb_cdddb050-55be-48e1-9a8e-df26711e4416/bond-cni-plugin/0.log" Jul 31 19:43:48.145104 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:48.145074 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p9mqb_cdddb050-55be-48e1-9a8e-df26711e4416/routeoverride-cni/0.log" Jul 31 19:43:48.170729 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:48.170502 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p9mqb_cdddb050-55be-48e1-9a8e-df26711e4416/whereabouts-cni-bincopy/0.log" Jul 31 19:43:48.195770 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:48.195738 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p9mqb_cdddb050-55be-48e1-9a8e-df26711e4416/whereabouts-cni/0.log" Jul 31 19:43:48.289096 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:48.289068 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-vbbdc_a9871869-ce7a-407a-8bf6-c1f871893ef9/kube-multus/0.log" Jul 31 19:43:48.314689 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:48.314614 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-h8n5q_ee8aa68f-72ff-4118-ad38-4da03b305feb/network-metrics-daemon/0.log" Jul 31 19:43:48.335186 ip-10-0-139-39 kubenswrapper[2565]: I0731 19:43:48.335159 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-h8n5q_ee8aa68f-72ff-4118-ad38-4da03b305feb/kube-rbac-proxy/0.log"